Jul 28 12:28:04.880462 ip-10-0-136-145 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 28 12:28:04.880476 ip-10-0-136-145 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 28 12:28:04.880485 ip-10-0-136-145 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 28 12:28:04.880804 ip-10-0-136-145 systemd[1]: Failed to start Kubernetes Kubelet. Jul 28 12:28:14.920185 ip-10-0-136-145 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 28 12:28:14.920200 ip-10-0-136-145 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8145e3606e9046b594db253c15981db5 -- Jul 28 12:30:13.305322 ip-10-0-136-145 systemd[1]: Starting Kubernetes Kubelet... Jul 28 12:30:13.694948 ip-10-0-136-145 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 12:30:13.694948 ip-10-0-136-145 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 12:30:13.694948 ip-10-0-136-145 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 12:30:13.694948 ip-10-0-136-145 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 12:30:13.694948 ip-10-0-136-145 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 12:30:13.695612 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.695012 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 12:30:13.699427 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699412 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 12:30:13.699427 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699426 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699430 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699435 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699439 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699442 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699445 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699448 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699450 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699453 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699456 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699458 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699461 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699463 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699466 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699468 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699470 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699478 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699481 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699484 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 12:30:13.699492 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699486 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699488 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699491 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699493 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699496 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699499 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699502 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699505 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699507 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699509 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699512 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699514 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699517 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699519 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699522 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699524 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699527 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699529 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699532 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699535 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 12:30:13.699921 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699538 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699540 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699542 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699544 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699547 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699549 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699551 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699553 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699557 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699560 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699562 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699564 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699567 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699570 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699572 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699575 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699577 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699580 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699582 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699584 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 12:30:13.700401 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699587 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699589 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699592 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699594 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699596 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699599 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699601 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699604 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699606 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699608 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699611 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699614 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699616 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699620 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699622 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699625 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699627 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699629 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699632 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699634 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 12:30:13.700869 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699637 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699639 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699642 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699646 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699649 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699652 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699655 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699658 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699660 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699662 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699665 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699667 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699670 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699672 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699675 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699679 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699682 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699684 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.699686 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 12:30:13.701389 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700368 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700375 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700378 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700381 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700383 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700387 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700389 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700392 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700394 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700397 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700399 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700402 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700405 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700408 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700412 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700416 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700419 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700422 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700424 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700427 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 12:30:13.701849 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700429 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700432 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700435 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700437 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700439 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700442 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700444 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700447 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700449 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700451 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700454 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700456 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700458 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700461 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700463 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700465 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700468 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700470 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700473 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 12:30:13.702352 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700477 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700480 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700483 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700487 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700490 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700494 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700496 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700499 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700501 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700504 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700507 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700509 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700511 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700514 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700516 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700518 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700521 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700523 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700526 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700528 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 12:30:13.702803 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700531 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700533 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700535 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700537 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700540 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700542 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700545 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700547 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700550 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700553 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700556 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700559 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700562 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700564 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700567 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700570 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700573 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700576 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700579 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 12:30:13.703316 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700581 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700583 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700586 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700588 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700591 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700593 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700595 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700598 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700601 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700603 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700606 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700608 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700610 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700613 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700615 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700617 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700620 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700622 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700624 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700627 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 12:30:13.703781 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.700629 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701007 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701012 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701015 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701019 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701040 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701045 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701049 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701072 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701075 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701078 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701081 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701084 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701087 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701089 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701092 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701094 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701097 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701100 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 12:30:13.704274 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701102 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701105 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701107 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701110 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701112 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701115 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701117 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701120 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701122 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701124 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701127 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701129 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701132 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701134 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701137 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701140 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701142 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701144 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701147 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 12:30:13.704761 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701150 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701153 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701155 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701160 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701162 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701165 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701167 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701170 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701172 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701175 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701177 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701179 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701182 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701185 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701187 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701190 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701192 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701195 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701197 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701200 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 12:30:13.705247 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701202 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701206 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701209 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701212 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701215 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701217 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701220 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701223 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701225 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701228 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701230 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701232 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701235 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701238 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701241 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701245 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701248 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701250 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701253 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 12:30:13.705731 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701255 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701257 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701260 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701262 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701265 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701267 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701270 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701272 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701274 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701277 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701279 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701282 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701284 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701287 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701289 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701292 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701294 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701296 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701299 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701302 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 12:30:13.706214 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701304 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701306 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701309 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701383 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701394 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701401 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701405 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701410 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701415 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701419 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701423 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701426 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701429 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701432 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701435 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701438 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701442 2573 flags.go:64] FLAG: --cgroup-root="" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701445 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701448 2573 flags.go:64] FLAG: --client-ca-file="" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701450 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701453 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701458 2573 flags.go:64] FLAG: --cluster-domain="" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701461 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701463 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 12:30:13.706693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701467 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701470 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701474 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701477 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701480 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701483 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701486 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701490 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701492 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701495 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701498 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701502 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701505 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701507 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701510 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701513 2573 flags.go:64] FLAG: --enable-server="true" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701519 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701524 2573 flags.go:64] FLAG: --event-burst="100" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701527 2573 flags.go:64] FLAG: --event-qps="50" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701530 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701532 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701535 2573 flags.go:64] FLAG: --eviction-hard="" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701539 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701542 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701544 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 12:30:13.707277 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701547 2573 flags.go:64] FLAG: --eviction-soft="" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701550 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701553 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701556 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701558 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701561 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701564 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701567 2573 flags.go:64] FLAG: --feature-gates="" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701571 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701573 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701576 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701579 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701582 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701585 2573 flags.go:64] FLAG: --help="false" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701588 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-136-145.ec2.internal" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701591 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701594 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701596 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701600 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701603 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701606 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701608 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701611 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 12:30:13.708082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701616 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701620 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701623 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701626 2573 flags.go:64] FLAG: --kube-reserved="" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701628 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701631 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701634 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701637 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701639 2573 flags.go:64] FLAG: --lock-file="" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701642 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701645 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701647 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701653 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701656 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701658 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701661 2573 flags.go:64] FLAG: --logging-format="text" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701663 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701668 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701671 2573 flags.go:64] FLAG: --manifest-url="" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701674 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701678 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701681 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701685 2573 flags.go:64] FLAG: --max-pods="110" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701687 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701690 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 12:30:13.709181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701693 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701695 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701698 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701701 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701704 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701710 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701713 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701717 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701720 2573 flags.go:64] FLAG: --pod-cidr="" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701723 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701729 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701731 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701735 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701737 2573 flags.go:64] FLAG: --port="10250" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701740 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701743 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06461731b9d8754fc" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701746 2573 flags.go:64] FLAG: --qos-reserved="" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701749 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701751 2573 flags.go:64] FLAG: --register-node="true" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701754 2573 flags.go:64] FLAG: --register-with-taints="" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701758 2573 flags.go:64] FLAG: --registry-burst="10" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701760 2573 flags.go:64] FLAG: --registry-qps="5" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701763 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701766 2573 flags.go:64] FLAG: --reserved-memory="" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701771 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 12:30:13.710281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701773 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701776 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701779 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701782 2573 flags.go:64] FLAG: --runonce="false" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701785 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701787 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701790 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701793 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701796 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701799 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701802 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701804 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701807 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701810 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701813 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701817 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701820 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701823 2573 flags.go:64] FLAG: --system-cgroups="" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701826 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701831 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701834 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701836 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701840 2573 flags.go:64] FLAG: --tls-min-version="" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701843 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701846 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 12:30:13.711274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701849 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701852 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701855 2573 flags.go:64] FLAG: --v="2" Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701859 2573 flags.go:64] FLAG: --version="false" Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701863 2573 flags.go:64] FLAG: --vmodule="" Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701866 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.701871 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701983 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701987 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701990 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701993 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701995 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.701998 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702001 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702004 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702006 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702009 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702011 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702014 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702018 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 12:30:13.712115 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702021 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702024 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702028 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702031 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702033 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702037 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702039 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702042 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702045 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702049 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702070 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702073 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702076 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702078 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702081 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702084 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702086 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702088 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702092 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 12:30:13.713086 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702095 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702097 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702100 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702102 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702105 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702108 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702110 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702113 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702116 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702118 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702120 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702123 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702125 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702140 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702143 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702146 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702148 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702151 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702153 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702156 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 12:30:13.713842 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702159 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702161 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702164 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702167 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702170 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702172 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702174 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702177 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702179 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702182 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702184 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702188 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702190 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702193 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702195 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702198 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702200 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702203 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702205 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702208 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 12:30:13.714388 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702210 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702212 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702214 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702217 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702220 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702223 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702226 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702229 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702231 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702234 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702236 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702238 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702241 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702243 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702246 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702248 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702251 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702253 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702255 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 12:30:13.714903 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702257 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702260 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702262 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702264 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702268 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702271 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702273 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.702275 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.702281 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.709692 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.709712 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.709736 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.709744 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709824 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709833 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 12:30:13.715374 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709838 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709843 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709847 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709854 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709860 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709865 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709870 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709875 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709879 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709883 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709888 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709892 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709897 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709901 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709905 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709908 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709913 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709917 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709921 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 12:30:13.715737 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709925 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709929 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709934 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709938 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709942 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709946 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709950 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709955 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709961 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709965 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709969 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709973 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709977 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709982 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709987 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709991 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.709995 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710001 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710007 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 12:30:13.716202 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710012 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710016 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710021 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710025 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710029 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710033 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710037 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710041 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710045 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710049 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710073 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710078 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710083 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710087 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710091 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710095 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710100 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710103 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710108 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710112 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 12:30:13.716643 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710116 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710133 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710138 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710142 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710146 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710150 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710154 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710158 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710162 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710166 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710170 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710174 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710178 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710182 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710186 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710190 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710194 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710199 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710203 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710207 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 12:30:13.717216 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710211 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710217 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710222 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710226 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710230 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710234 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710238 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710242 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710247 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710251 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710255 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710259 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710264 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710268 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710272 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710277 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710280 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710284 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 12:30:13.717689 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710288 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.710295 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710460 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710469 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710473 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710477 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710481 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710486 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710490 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710494 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710498 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710503 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710509 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710516 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710522 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 12:30:13.718198 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710526 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710532 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710537 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710542 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710547 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710551 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710555 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710559 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710563 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710568 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710572 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710575 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710579 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710583 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710587 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710592 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710596 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710600 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710604 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 12:30:13.718551 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710608 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710613 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710617 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710621 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710625 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710629 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710633 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710637 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710641 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710644 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710648 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710652 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710657 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710661 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710665 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710669 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710673 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710677 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710681 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710684 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 12:30:13.718988 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710688 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710693 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710697 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710701 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710704 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710708 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710712 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710716 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710720 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710724 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710729 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710733 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710737 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710741 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710745 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710749 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710753 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710757 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710760 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710764 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 12:30:13.719484 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710768 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710772 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710776 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710780 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710784 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710788 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710792 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710796 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710800 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710804 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710808 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710812 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710816 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710820 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710824 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710828 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710832 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710836 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710840 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710844 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 12:30:13.719938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710848 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 12:30:13.720424 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710853 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 12:30:13.720424 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710860 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 12:30:13.720424 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710865 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 12:30:13.720424 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710869 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 12:30:13.720424 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710873 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 12:30:13.720424 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:13.710877 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 12:30:13.720424 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.710884 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 12:30:13.720424 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.711022 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 12:30:13.720424 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.715717 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 12:30:13.720424 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.716998 2573 server.go:1019] "Starting client certificate rotation" Jul 28 12:30:13.720424 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.717087 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 12:30:13.720424 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.717118 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 12:30:13.720424 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.720104 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 12:30:13.723738 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.723721 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 12:30:13.731976 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.731960 2573 log.go:25] "Validated CRI v1 runtime API" Jul 28 12:30:13.733911 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.733897 2573 log.go:25] "Validated CRI v1 image API" Jul 28 12:30:13.734726 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.734712 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 12:30:13.739244 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.739226 2573 fs.go:135] Filesystem UUIDs: map[6599e7cf-ab1c-4875-acc6-b68a65848802:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d5e7a40c-6227-4e85-8a2e-dff724124ad6:/dev/nvme0n1p4] Jul 28 12:30:13.739322 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.739244 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 12:30:13.741330 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.741310 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 12:30:13.745604 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.745495 2573 manager.go:216] Machine: {Timestamp:2026-07-28 12:30:13.744259148 +0000 UTC m=+0.319933349 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3181465 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20df28b3e8f5c86608327cb2dcc833 SystemUUID:ec20df28-b3e8-f5c8-6608-327cb2dcc833 BootID:8145e360-6e90-46b5-94db-253c15981db5 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7e:b9:16:4c:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7e:b9:16:4c:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:57:1a:fa:00:30 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 12:30:13.745604 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.745600 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 12:30:13.745699 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.745670 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 12:30:13.748507 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.748486 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 12:30:13.748639 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.748508 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-145.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 12:30:13.749679 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.749669 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 12:30:13.749710 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.749682 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 12:30:13.749710 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.749690 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 12:30:13.750279 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.750269 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 12:30:13.751220 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.751210 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 12:30:13.752752 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.752742 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 28 12:30:13.752872 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.752863 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 12:30:13.752900 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.752891 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 28 12:30:13.752928 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.752912 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 12:30:13.752964 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.752956 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 12:30:13.753000 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.752971 2573 kubelet.go:398] "Adding apiserver pod source" Jul 28 12:30:13.753000 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.752995 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 12:30:13.753489 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.753472 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 12:30:13.753489 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.753490 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 12:30:13.755604 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.755583 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 12:30:13.757451 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.757438 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 12:30:13.757507 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.757468 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 12:30:13.760332 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.760316 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 12:30:13.760390 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.760338 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 12:30:13.760390 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.760344 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 12:30:13.760390 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.760350 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 12:30:13.760390 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.760359 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 12:30:13.760390 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.760366 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 12:30:13.760390 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.760375 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 12:30:13.760390 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.760380 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 12:30:13.760390 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.760387 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 12:30:13.760390 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.760392 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 12:30:13.760621 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.760406 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 12:30:13.760621 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.760411 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 12:30:13.761775 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.761763 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 12:30:13.761775 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.761775 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 12:30:13.762585 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:13.762562 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-145.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 12:30:13.762671 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:13.762654 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 12:30:13.764788 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.764771 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r9c7t" Jul 28 12:30:13.764838 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.764827 2573 server.go:1268] "Started kubelet" Jul 28 12:30:13.764956 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.764912 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 12:30:13.765010 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.764976 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 12:30:13.765126 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.765077 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 12:30:13.765700 ip-10-0-136-145 systemd[1]: Started Kubernetes Kubelet. Jul 28 12:30:13.766761 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.766745 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 12:30:13.768031 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.768016 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 28 12:30:13.770090 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:13.770045 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 12:30:13.772404 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.772385 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r9c7t" Jul 28 12:30:13.772517 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.772504 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-145.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 12:30:13.773438 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:13.772546 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-145.ec2.internal.18c673f893a007f0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-145.ec2.internal,UID:ip-10-0-136-145.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-145.ec2.internal,},FirstTimestamp:2026-07-28 12:30:13.764802544 +0000 UTC m=+0.340476742,LastTimestamp:2026-07-28 12:30:13.764802544 +0000 UTC m=+0.340476742,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-145.ec2.internal,}" Jul 28 12:30:13.774388 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.774371 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 12:30:13.774388 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.774375 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 12:30:13.774525 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:13.774509 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:13.775162 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.775143 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 12:30:13.775162 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.775165 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 12:30:13.775512 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.775408 2573 factory.go:55] Registering systemd factory Jul 28 12:30:13.775512 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.775428 2573 factory.go:223] Registration of the systemd container factory successfully Jul 28 12:30:13.775615 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.775514 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 28 12:30:13.775615 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.775521 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 28 12:30:13.775709 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.775663 2573 factory.go:153] Registering CRI-O factory Jul 28 12:30:13.775709 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.775676 2573 factory.go:223] Registration of the crio container factory successfully Jul 28 12:30:13.775804 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.775716 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 12:30:13.775804 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.775742 2573 factory.go:103] Registering Raw factory Jul 28 12:30:13.775804 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.775758 2573 manager.go:1196] Started watching for new ooms in manager Jul 28 12:30:13.775804 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.775778 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 12:30:13.776684 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.776669 2573 manager.go:349] Starting recovery of all containers Jul 28 12:30:13.777672 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:13.777646 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-145.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 12:30:13.777823 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:13.777785 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 12:30:13.789380 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.789241 2573 manager.go:354] Recovery completed Jul 28 12:30:13.794981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.794966 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:30:13.796726 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.796712 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:30:13.796794 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.796739 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:30:13.796794 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.796750 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:30:13.797368 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.797352 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 12:30:13.797436 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.797370 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 12:30:13.797436 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.797388 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 28 12:30:13.797537 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.797356 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:30:13.798175 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.798148 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:30:13.798242 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.798190 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:30:13.798242 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.798205 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:30:13.798988 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.798976 2573 policy_none.go:49] "None policy: Start" Jul 28 12:30:13.799026 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.798992 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 12:30:13.799026 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.799002 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 12:30:13.799970 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.799960 2573 policy_none.go:47] "Start" Jul 28 12:30:13.843636 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.843621 2573 manager.go:337] "Starting Device Plugin manager" Jul 28 12:30:13.843721 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:13.843649 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 12:30:13.843721 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.843659 2573 server.go:87] "Starting device plugin registration server" Jul 28 12:30:13.843882 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.843869 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 12:30:13.843935 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.843883 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 12:30:13.843978 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.843958 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 12:30:13.844086 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.844070 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 12:30:13.844086 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.844085 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 12:30:13.844632 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:13.844608 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 12:30:13.844694 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:13.844659 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:13.943519 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.943482 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 12:30:13.945501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.943977 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:30:13.945501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.944844 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 12:30:13.945501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.944880 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 12:30:13.945501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.944898 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 12:30:13.945501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.944902 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:30:13.945501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.944929 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:30:13.945501 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:13.944936 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 12:30:13.945501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.944944 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:30:13.945501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.944976 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-145.ec2.internal" Jul 28 12:30:13.947025 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.947007 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 12:30:13.950880 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:13.950865 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-145.ec2.internal" Jul 28 12:30:13.950948 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:13.950885 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-145.ec2.internal\": node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:14.000194 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:14.000173 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:14.045259 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.045229 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-145.ec2.internal"] Jul 28 12:30:14.045321 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.045298 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:30:14.046136 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.046123 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:30:14.046197 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.046147 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:30:14.046197 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.046168 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:30:14.049831 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.049818 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:30:14.049954 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.049940 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.050011 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.049974 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:30:14.050473 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.050460 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:30:14.050552 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.050473 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:30:14.050552 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.050483 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:30:14.050552 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.050494 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:30:14.050552 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.050498 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:30:14.050552 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.050508 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:30:14.052563 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.052546 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.052641 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.052575 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:30:14.053279 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.053262 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:30:14.053375 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.053287 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:30:14.053375 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.053302 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:30:14.077323 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.077306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62537a11327111f64b516ac280dab601-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal\" (UID: \"62537a11327111f64b516ac280dab601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.077414 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.077332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62537a11327111f64b516ac280dab601-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal\" (UID: \"62537a11327111f64b516ac280dab601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.077414 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.077349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4541c39d7375e0d14063052c99c9a71f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-145.ec2.internal\" (UID: \"4541c39d7375e0d14063052c99c9a71f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.083743 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:14.083728 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-145.ec2.internal\" not found" node="ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.087076 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:14.087045 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-145.ec2.internal\" not found" node="ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.101277 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:14.101259 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:14.178085 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.178064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62537a11327111f64b516ac280dab601-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal\" (UID: \"62537a11327111f64b516ac280dab601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.178170 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.178087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62537a11327111f64b516ac280dab601-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal\" (UID: \"62537a11327111f64b516ac280dab601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.178170 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.178125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4541c39d7375e0d14063052c99c9a71f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-145.ec2.internal\" (UID: \"4541c39d7375e0d14063052c99c9a71f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.178261 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.178171 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4541c39d7375e0d14063052c99c9a71f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-145.ec2.internal\" (UID: \"4541c39d7375e0d14063052c99c9a71f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.178261 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.178230 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62537a11327111f64b516ac280dab601-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal\" (UID: \"62537a11327111f64b516ac280dab601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.178330 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.178259 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62537a11327111f64b516ac280dab601-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal\" (UID: \"62537a11327111f64b516ac280dab601\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.202192 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:14.202150 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:14.302965 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:14.302945 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:14.386513 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.386486 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.389744 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.389728 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-145.ec2.internal" Jul 28 12:30:14.403793 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:14.403777 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:14.504360 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:14.504298 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:14.578405 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.578382 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 12:30:14.605114 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:14.605095 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:14.705635 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:14.705616 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:14.716792 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.716772 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 12:30:14.716917 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.716901 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 12:30:14.716953 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.716930 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 12:30:14.777442 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.777365 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 12:25:15 +0000 UTC" deadline="2027-12-25 02:04:53.850702643 +0000 UTC" Jul 28 12:30:14.777442 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.777397 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12349h34m39.073308701s" Jul 28 12:30:14.805768 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:14.805737 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:14.890946 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:14.890914 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 12:30:14.906494 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:14.906474 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:15.007187 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:15.007161 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:15.031427 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:15.031368 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4541c39d7375e0d14063052c99c9a71f.slice/crio-a72b148ff7f5f866d104cf6bd99e0ccc6bcf021b13d5e591cf94bbe6b3370924 WatchSource:0}: Error finding container a72b148ff7f5f866d104cf6bd99e0ccc6bcf021b13d5e591cf94bbe6b3370924: Status 404 returned error can't find the container with id a72b148ff7f5f866d104cf6bd99e0ccc6bcf021b13d5e591cf94bbe6b3370924 Jul 28 12:30:15.031828 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:15.031801 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62537a11327111f64b516ac280dab601.slice/crio-6bc4fc697f385ab6ffeb12b1ebc2caf2df302aca062eb3b699d28056e77a33ee WatchSource:0}: Error finding container 6bc4fc697f385ab6ffeb12b1ebc2caf2df302aca062eb3b699d28056e77a33ee: Status 404 returned error can't find the container with id 6bc4fc697f385ab6ffeb12b1ebc2caf2df302aca062eb3b699d28056e77a33ee Jul 28 12:30:15.036224 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:15.036209 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 12:30:15.107608 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:15.107586 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-145.ec2.internal\" not found" Jul 28 12:30:15.143611 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:15.143598 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 12:30:16.491561 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:16.491547 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal" Jul 28 12:30:16.502893 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:16.502877 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 12:30:16.503774 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:16.503763 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-145.ec2.internal" Jul 28 12:30:16.512448 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:16.512428 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 12:30:16.917375 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:16.917345 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 12:30:17.071525 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.071498 2573 apiserver.go:52] "Watching apiserver" Jul 28 12:30:17.086307 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.086280 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 12:30:17.086702 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.086676 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cg9km","kube-system/kube-apiserver-proxy-ip-10-0-136-145.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf","openshift-cluster-node-tuning-operator/tuned-5jzt2","openshift-dns/node-resolver-wd7wp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal","openshift-multus/multus-additional-cni-plugins-strvp","openshift-multus/multus-w262h","kube-system/konnectivity-agent-n88bh","openshift-image-registry/node-ca-vthpv","openshift-multus/network-metrics-daemon-csfdh","openshift-network-diagnostics/network-check-target-564mm","openshift-network-operator/iptables-alerter-54zx6","openshift-ovn-kubernetes/ovnkube-node-kfk7p"] Jul 28 12:30:17.089229 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.089193 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.091523 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.091210 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 12:30:17.091991 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.091973 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-54zx6" Jul 28 12:30:17.092814 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.092793 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 12:30:17.092995 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.092980 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 12:30:17.093174 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.093160 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bzb22\"" Jul 28 12:30:17.093362 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.093347 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 12:30:17.093713 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.093696 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 12:30:17.093862 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.093849 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 12:30:17.097180 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.097162 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 12:30:17.097314 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.097295 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 12:30:17.097887 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.097602 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.097887 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.097718 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.099169 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.098030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wd7wp" Jul 28 12:30:17.099169 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.098836 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vqb5p\"" Jul 28 12:30:17.099169 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.098839 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 12:30:17.100512 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.100494 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.102072 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.100732 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 12:30:17.102072 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.100886 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 12:30:17.102072 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.101343 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-whbjj\"" Jul 28 12:30:17.102072 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.101473 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 12:30:17.102072 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.101658 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 12:30:17.102072 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.101742 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 12:30:17.102072 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.101842 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mlvwk\"" Jul 28 12:30:17.102072 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.101923 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 12:30:17.102501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.102324 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xmdhq\"" Jul 28 12:30:17.102501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.102482 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 12:30:17.103624 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.102975 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n88bh" Jul 28 12:30:17.103709 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.103632 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 12:30:17.104318 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.103877 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8678p\"" Jul 28 12:30:17.104318 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.103955 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 12:30:17.104318 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.104037 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 12:30:17.104318 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.104163 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 12:30:17.104826 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.104807 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 12:30:17.104901 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.104866 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 12:30:17.105233 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105216 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:17.105337 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.105314 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:17.105425 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105381 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 12:30:17.105549 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105529 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-device-dir\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.105658 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105564 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-host\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.105658 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105588 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvdm\" (UniqueName: \"kubernetes.io/projected/16de775e-beb3-4834-9a10-724db2f7003b-kube-api-access-5cvdm\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.105658 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105613 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h5tq\" (UniqueName: \"kubernetes.io/projected/8325bc39-4cb3-49bb-b803-18c2ab4a24e7-kube-api-access-5h5tq\") pod \"node-resolver-wd7wp\" (UID: \"8325bc39-4cb3-49bb-b803-18c2ab4a24e7\") " pod="openshift-dns/node-resolver-wd7wp" Jul 28 12:30:17.105658 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105637 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlhv\" (UniqueName: \"kubernetes.io/projected/32eea5ab-06de-4cc0-ab3d-16c855389a86-kube-api-access-lxlhv\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.105856 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfpf2\" (UniqueName: \"kubernetes.io/projected/51971a1f-fb2e-481f-8a79-b80660f239fd-kube-api-access-kfpf2\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.105856 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105696 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9fba58b4-2405-4d1c-8413-219e279f973a-host-slash\") pod \"iptables-alerter-54zx6\" (UID: \"9fba58b4-2405-4d1c-8413-219e279f973a\") " pod="openshift-network-operator/iptables-alerter-54zx6" Jul 28 12:30:17.105856 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105728 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.105856 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105754 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-sys-fs\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.105856 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105776 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-kubernetes\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.105856 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105793 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 12:30:17.105856 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105799 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-sysctl-d\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.105856 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105835 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32eea5ab-06de-4cc0-ab3d-16c855389a86-tuning-conf-dir\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105860 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105885 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-lib-modules\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-var-lib-kubelet\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105953 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32eea5ab-06de-4cc0-ab3d-16c855389a86-system-cni-dir\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.105984 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32eea5ab-06de-4cc0-ab3d-16c855389a86-cnibin\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106010 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gj5rb\"" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106021 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32eea5ab-06de-4cc0-ab3d-16c855389a86-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106056 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-systemd\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106092 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp9mz\" (UniqueName: \"kubernetes.io/projected/9fba58b4-2405-4d1c-8413-219e279f973a-kube-api-access-kp9mz\") pod \"iptables-alerter-54zx6\" (UID: \"9fba58b4-2405-4d1c-8413-219e279f973a\") " pod="openshift-network-operator/iptables-alerter-54zx6" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106116 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-registration-dir\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-sysconfig\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106161 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-sysctl-conf\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106183 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-run\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16de775e-beb3-4834-9a10-724db2f7003b-tmp\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.106248 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106226 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32eea5ab-06de-4cc0-ab3d-16c855389a86-cni-binary-copy\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.107076 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106274 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-sys\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.107076 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106309 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8325bc39-4cb3-49bb-b803-18c2ab4a24e7-hosts-file\") pod \"node-resolver-wd7wp\" (UID: \"8325bc39-4cb3-49bb-b803-18c2ab4a24e7\") " pod="openshift-dns/node-resolver-wd7wp" Jul 28 12:30:17.107076 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106346 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32eea5ab-06de-4cc0-ab3d-16c855389a86-os-release\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.107076 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-socket-dir\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.107076 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-modprobe-d\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.107076 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16de775e-beb3-4834-9a10-724db2f7003b-etc-tuned\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.107076 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8325bc39-4cb3-49bb-b803-18c2ab4a24e7-tmp-dir\") pod \"node-resolver-wd7wp\" (UID: \"8325bc39-4cb3-49bb-b803-18c2ab4a24e7\") " pod="openshift-dns/node-resolver-wd7wp" Jul 28 12:30:17.107076 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106502 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32eea5ab-06de-4cc0-ab3d-16c855389a86-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.107076 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.106525 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9fba58b4-2405-4d1c-8413-219e279f973a-iptables-alerter-script\") pod \"iptables-alerter-54zx6\" (UID: \"9fba58b4-2405-4d1c-8413-219e279f973a\") " pod="openshift-network-operator/iptables-alerter-54zx6" Jul 28 12:30:17.107665 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.107647 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w262h" Jul 28 12:30:17.110063 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.110040 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 12:30:17.111671 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.110280 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vthpv" Jul 28 12:30:17.111671 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.110313 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5chlh\"" Jul 28 12:30:17.111671 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.110747 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 12:30:17.112494 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.112474 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:17.112582 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.112538 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:17.114567 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.114547 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4j87x\"" Jul 28 12:30:17.114657 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.114636 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:17.114716 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.114702 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:17.114772 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.114747 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 12:30:17.121432 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.121394 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 12:30:17.122270 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.121684 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 12:30:17.194071 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.194052 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 12:30:17.207081 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-device-dir\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.207177 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207093 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/679a7244-b847-4ffc-b0c1-6c9c023eb440-ovnkube-script-lib\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.207177 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15335f10-5396-417f-9436-0b8abd1664be-cni-binary-copy\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.207177 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-run-k8s-cni-cncf-io\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.207285 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-etc-openvswitch\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.207285 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207197 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-cni-netd\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.207285 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/679a7244-b847-4ffc-b0c1-6c9c023eb440-ovnkube-config\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.207285 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207238 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-os-release\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.207285 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207257 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:17.207285 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207281 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.207511 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207292 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-device-dir\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.207511 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207305 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-sysctl-d\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.207511 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32eea5ab-06de-4cc0-ab3d-16c855389a86-tuning-conf-dir\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.207511 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-run-netns\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.207511 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-node-log\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.207511 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207390 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/679a7244-b847-4ffc-b0c1-6c9c023eb440-ovn-node-metrics-cert\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.207511 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207413 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.207511 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f2af6702-608a-497a-bcc5-d7eac4802553-serviceca\") pod \"node-ca-vthpv\" (UID: \"f2af6702-608a-497a-bcc5-d7eac4802553\") " pod="openshift-image-registry/node-ca-vthpv" Jul 28 12:30:17.207511 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207492 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-lib-modules\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.207511 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-var-lib-kubelet\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207535 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-sysctl-d\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207542 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32eea5ab-06de-4cc0-ab3d-16c855389a86-cnibin\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207604 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-run-ovn\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207636 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-multus-socket-dir-parent\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207674 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-lib-modules\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207694 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32eea5ab-06de-4cc0-ab3d-16c855389a86-tuning-conf-dir\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-var-lib-cni-bin\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207728 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-registration-dir\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207741 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-var-lib-kubelet\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207755 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-run\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207777 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-registration-dir\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207816 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32eea5ab-06de-4cc0-ab3d-16c855389a86-cnibin\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16de775e-beb3-4834-9a10-724db2f7003b-tmp\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207863 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-run\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.207921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207898 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-systemd-units\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.207970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-run-netns\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208000 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-hostroot\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-sys\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208039 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8325bc39-4cb3-49bb-b803-18c2ab4a24e7-hosts-file\") pod \"node-resolver-wd7wp\" (UID: \"8325bc39-4cb3-49bb-b803-18c2ab4a24e7\") " pod="openshift-dns/node-resolver-wd7wp" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208093 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-sys\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208097 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32eea5ab-06de-4cc0-ab3d-16c855389a86-os-release\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208112 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8325bc39-4cb3-49bb-b803-18c2ab4a24e7-hosts-file\") pod \"node-resolver-wd7wp\" (UID: \"8325bc39-4cb3-49bb-b803-18c2ab4a24e7\") " pod="openshift-dns/node-resolver-wd7wp" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208134 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbf8d227-a317-4952-a0b2-7ca6d602d804-kubelet-config\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208134 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208162 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32eea5ab-06de-4cc0-ab3d-16c855389a86-os-release\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208191 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-multus-conf-dir\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-socket-dir\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-modprobe-d\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8325bc39-4cb3-49bb-b803-18c2ab4a24e7-tmp-dir\") pod \"node-resolver-wd7wp\" (UID: \"8325bc39-4cb3-49bb-b803-18c2ab4a24e7\") " pod="openshift-dns/node-resolver-wd7wp" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208313 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-socket-dir\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-slash\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208356 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-modprobe-d\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.208619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-var-lib-openvswitch\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208428 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-run-multus-certs\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208475 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bdfw\" (UniqueName: \"kubernetes.io/projected/a4868cb1-759e-4f1a-a614-165690edf4a3-kube-api-access-7bdfw\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208503 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-host\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208526 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvdm\" (UniqueName: \"kubernetes.io/projected/16de775e-beb3-4834-9a10-724db2f7003b-kube-api-access-5cvdm\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208550 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h5tq\" (UniqueName: \"kubernetes.io/projected/8325bc39-4cb3-49bb-b803-18c2ab4a24e7-kube-api-access-5h5tq\") pod \"node-resolver-wd7wp\" (UID: \"8325bc39-4cb3-49bb-b803-18c2ab4a24e7\") " pod="openshift-dns/node-resolver-wd7wp" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208574 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlhv\" (UniqueName: \"kubernetes.io/projected/32eea5ab-06de-4cc0-ab3d-16c855389a86-kube-api-access-lxlhv\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208589 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8325bc39-4cb3-49bb-b803-18c2ab4a24e7-tmp-dir\") pod \"node-resolver-wd7wp\" (UID: \"8325bc39-4cb3-49bb-b803-18c2ab4a24e7\") " pod="openshift-dns/node-resolver-wd7wp" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208595 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2af6702-608a-497a-bcc5-d7eac4802553-host\") pod \"node-ca-vthpv\" (UID: \"f2af6702-608a-497a-bcc5-d7eac4802553\") " pod="openshift-image-registry/node-ca-vthpv" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208618 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfpf2\" (UniqueName: \"kubernetes.io/projected/51971a1f-fb2e-481f-8a79-b80660f239fd-kube-api-access-kfpf2\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208640 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9fba58b4-2405-4d1c-8413-219e279f973a-host-slash\") pod \"iptables-alerter-54zx6\" (UID: \"9fba58b4-2405-4d1c-8413-219e279f973a\") " pod="openshift-network-operator/iptables-alerter-54zx6" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208676 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/679a7244-b847-4ffc-b0c1-6c9c023eb440-env-overrides\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208697 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-system-cni-dir\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208717 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmwhj\" (UniqueName: \"kubernetes.io/projected/f2af6702-608a-497a-bcc5-d7eac4802553-kube-api-access-pmwhj\") pod \"node-ca-vthpv\" (UID: \"f2af6702-608a-497a-bcc5-d7eac4802553\") " pod="openshift-image-registry/node-ca-vthpv" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-sys-fs\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-kubernetes\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208783 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-kubelet\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.209410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208825 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-run-systemd\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208847 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-run-openvswitch\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208890 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-host\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208935 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32eea5ab-06de-4cc0-ab3d-16c855389a86-system-cni-dir\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32eea5ab-06de-4cc0-ab3d-16c855389a86-system-cni-dir\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208968 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32eea5ab-06de-4cc0-ab3d-16c855389a86-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.208994 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-cni-bin\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbf8d227-a317-4952-a0b2-7ca6d602d804-dbus\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-multus-cni-dir\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209059 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-systemd\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209092 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9mz\" (UniqueName: \"kubernetes.io/projected/9fba58b4-2405-4d1c-8413-219e279f973a-kube-api-access-kp9mz\") pod \"iptables-alerter-54zx6\" (UID: \"9fba58b4-2405-4d1c-8413-219e279f973a\") " pod="openshift-network-operator/iptables-alerter-54zx6" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209115 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp9xh\" (UniqueName: \"kubernetes.io/projected/679a7244-b847-4ffc-b0c1-6c9c023eb440-kube-api-access-wp9xh\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-cnibin\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209167 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-var-lib-cni-multus\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209189 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-etc-kubernetes\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2ldm\" (UniqueName: \"kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm\") pod \"network-check-target-564mm\" (UID: \"42398dea-bac1-4abf-accc-beca393738af\") " pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:17.210151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-sysconfig\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209500 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-sysconfig\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209515 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-kubernetes\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-sysctl-conf\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209583 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209593 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51971a1f-fb2e-481f-8a79-b80660f239fd-sys-fs\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209634 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-systemd\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209669 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9fba58b4-2405-4d1c-8413-219e279f973a-host-slash\") pod \"iptables-alerter-54zx6\" (UID: \"9fba58b4-2405-4d1c-8413-219e279f973a\") " pod="openshift-network-operator/iptables-alerter-54zx6" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32eea5ab-06de-4cc0-ab3d-16c855389a86-cni-binary-copy\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209776 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-log-socket\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209817 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16de775e-beb3-4834-9a10-724db2f7003b-etc-sysctl-conf\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209849 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7d55ba4-f5a8-48fd-8c9f-932ceaedc498-konnectivity-ca\") pod \"konnectivity-agent-n88bh\" (UID: \"c7d55ba4-f5a8-48fd-8c9f-932ceaedc498\") " pod="kube-system/konnectivity-agent-n88bh" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-var-lib-kubelet\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15335f10-5396-417f-9436-0b8abd1664be-multus-daemon-config\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.209968 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.210012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.210027 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32eea5ab-06de-4cc0-ab3d-16c855389a86-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.210924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.210038 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7d55ba4-f5a8-48fd-8c9f-932ceaedc498-agent-certs\") pod \"konnectivity-agent-n88bh\" (UID: \"c7d55ba4-f5a8-48fd-8c9f-932ceaedc498\") " pod="kube-system/konnectivity-agent-n88bh" Jul 28 12:30:17.211689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.210065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16de775e-beb3-4834-9a10-724db2f7003b-etc-tuned\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.211689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.210092 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32eea5ab-06de-4cc0-ab3d-16c855389a86-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.211689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.210116 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9fba58b4-2405-4d1c-8413-219e279f973a-iptables-alerter-script\") pod \"iptables-alerter-54zx6\" (UID: \"9fba58b4-2405-4d1c-8413-219e279f973a\") " pod="openshift-network-operator/iptables-alerter-54zx6" Jul 28 12:30:17.211689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.210182 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32eea5ab-06de-4cc0-ab3d-16c855389a86-cni-binary-copy\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.211689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.210218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvdp\" (UniqueName: \"kubernetes.io/projected/15335f10-5396-417f-9436-0b8abd1664be-kube-api-access-dcvdp\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.211689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.210544 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32eea5ab-06de-4cc0-ab3d-16c855389a86-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.211689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.210607 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9fba58b4-2405-4d1c-8413-219e279f973a-iptables-alerter-script\") pod \"iptables-alerter-54zx6\" (UID: \"9fba58b4-2405-4d1c-8413-219e279f973a\") " pod="openshift-network-operator/iptables-alerter-54zx6" Jul 28 12:30:17.213669 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.213646 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16de775e-beb3-4834-9a10-724db2f7003b-etc-tuned\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.213767 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.213698 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16de775e-beb3-4834-9a10-724db2f7003b-tmp\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.220825 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.220804 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h5tq\" (UniqueName: \"kubernetes.io/projected/8325bc39-4cb3-49bb-b803-18c2ab4a24e7-kube-api-access-5h5tq\") pod \"node-resolver-wd7wp\" (UID: \"8325bc39-4cb3-49bb-b803-18c2ab4a24e7\") " pod="openshift-dns/node-resolver-wd7wp" Jul 28 12:30:17.223776 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.223744 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlhv\" (UniqueName: \"kubernetes.io/projected/32eea5ab-06de-4cc0-ab3d-16c855389a86-kube-api-access-lxlhv\") pod \"multus-additional-cni-plugins-strvp\" (UID: \"32eea5ab-06de-4cc0-ab3d-16c855389a86\") " pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.224239 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.224217 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfpf2\" (UniqueName: \"kubernetes.io/projected/51971a1f-fb2e-481f-8a79-b80660f239fd-kube-api-access-kfpf2\") pod \"aws-ebs-csi-driver-node-69gmf\" (UID: \"51971a1f-fb2e-481f-8a79-b80660f239fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.224582 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.224565 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvdm\" (UniqueName: \"kubernetes.io/projected/16de775e-beb3-4834-9a10-724db2f7003b-kube-api-access-5cvdm\") pod \"tuned-5jzt2\" (UID: \"16de775e-beb3-4834-9a10-724db2f7003b\") " pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.225526 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.225486 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9mz\" (UniqueName: \"kubernetes.io/projected/9fba58b4-2405-4d1c-8413-219e279f973a-kube-api-access-kp9mz\") pod \"iptables-alerter-54zx6\" (UID: \"9fba58b4-2405-4d1c-8413-219e279f973a\") " pod="openshift-network-operator/iptables-alerter-54zx6" Jul 28 12:30:17.255870 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.255835 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wqctp" Jul 28 12:30:17.266583 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.266528 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-145.ec2.internal" event={"ID":"4541c39d7375e0d14063052c99c9a71f","Type":"ContainerStarted","Data":"a72b148ff7f5f866d104cf6bd99e0ccc6bcf021b13d5e591cf94bbe6b3370924"} Jul 28 12:30:17.267275 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.267256 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wqctp" Jul 28 12:30:17.267727 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.267706 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal" event={"ID":"62537a11327111f64b516ac280dab601","Type":"ContainerStarted","Data":"6bc4fc697f385ab6ffeb12b1ebc2caf2df302aca062eb3b699d28056e77a33ee"} Jul 28 12:30:17.311486 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bdfw\" (UniqueName: \"kubernetes.io/projected/a4868cb1-759e-4f1a-a614-165690edf4a3-kube-api-access-7bdfw\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:17.311486 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2af6702-608a-497a-bcc5-d7eac4802553-host\") pod \"node-ca-vthpv\" (UID: \"f2af6702-608a-497a-bcc5-d7eac4802553\") " pod="openshift-image-registry/node-ca-vthpv" Jul 28 12:30:17.311716 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/679a7244-b847-4ffc-b0c1-6c9c023eb440-env-overrides\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.311716 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311622 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-system-cni-dir\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.311716 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311651 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwhj\" (UniqueName: \"kubernetes.io/projected/f2af6702-608a-497a-bcc5-d7eac4802553-kube-api-access-pmwhj\") pod \"node-ca-vthpv\" (UID: \"f2af6702-608a-497a-bcc5-d7eac4802553\") " pod="openshift-image-registry/node-ca-vthpv" Jul 28 12:30:17.311716 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311655 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2af6702-608a-497a-bcc5-d7eac4802553-host\") pod \"node-ca-vthpv\" (UID: \"f2af6702-608a-497a-bcc5-d7eac4802553\") " pod="openshift-image-registry/node-ca-vthpv" Jul 28 12:30:17.311716 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311673 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-kubelet\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.311716 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311693 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-run-systemd\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.311902 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311740 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-run-openvswitch\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.311902 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311752 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-system-cni-dir\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.311902 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-cni-bin\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.311902 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311803 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-run-openvswitch\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.311902 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbf8d227-a317-4952-a0b2-7ca6d602d804-dbus\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:17.311902 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311838 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-run-systemd\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.311902 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-multus-cni-dir\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.311902 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9xh\" (UniqueName: \"kubernetes.io/projected/679a7244-b847-4ffc-b0c1-6c9c023eb440-kube-api-access-wp9xh\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.311902 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311875 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-cni-bin\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311913 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-cnibin\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311907 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-kubelet\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311974 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-var-lib-cni-multus\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.311944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-var-lib-cni-multus\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312000 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-etc-kubernetes\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ldm\" (UniqueName: \"kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm\") pod \"network-check-target-564mm\" (UID: \"42398dea-bac1-4abf-accc-beca393738af\") " pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312033 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-log-socket\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7d55ba4-f5a8-48fd-8c9f-932ceaedc498-konnectivity-ca\") pod \"konnectivity-agent-n88bh\" (UID: \"c7d55ba4-f5a8-48fd-8c9f-932ceaedc498\") " pod="kube-system/konnectivity-agent-n88bh" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312061 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-multus-cni-dir\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-var-lib-kubelet\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312065 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-cnibin\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312105 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbf8d227-a317-4952-a0b2-7ca6d602d804-dbus\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312113 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15335f10-5396-417f-9436-0b8abd1664be-multus-daemon-config\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312141 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-etc-kubernetes\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312149 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312191 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7d55ba4-f5a8-48fd-8c9f-932ceaedc498-agent-certs\") pod \"konnectivity-agent-n88bh\" (UID: \"c7d55ba4-f5a8-48fd-8c9f-932ceaedc498\") " pod="kube-system/konnectivity-agent-n88bh" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312198 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-var-lib-kubelet\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvdp\" (UniqueName: \"kubernetes.io/projected/15335f10-5396-417f-9436-0b8abd1664be-kube-api-access-dcvdp\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312248 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-log-socket\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/679a7244-b847-4ffc-b0c1-6c9c023eb440-ovnkube-script-lib\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312266 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312299 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15335f10-5396-417f-9436-0b8abd1664be-cni-binary-copy\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-run-k8s-cni-cncf-io\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-etc-openvswitch\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-cni-netd\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312427 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/679a7244-b847-4ffc-b0c1-6c9c023eb440-ovnkube-config\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312473 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-os-release\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-run-netns\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-node-log\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/679a7244-b847-4ffc-b0c1-6c9c023eb440-ovn-node-metrics-cert\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.312689 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f2af6702-608a-497a-bcc5-d7eac4802553-serviceca\") pod \"node-ca-vthpv\" (UID: \"f2af6702-608a-497a-bcc5-d7eac4802553\") " pod="openshift-image-registry/node-ca-vthpv" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-run-ovn\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312741 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312763 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-multus-socket-dir-parent\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312784 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-var-lib-cni-bin\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312801 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/679a7244-b847-4ffc-b0c1-6c9c023eb440-ovnkube-script-lib\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312810 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-systemd-units\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312832 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-run-netns\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-hostroot\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbf8d227-a317-4952-a0b2-7ca6d602d804-kubelet-config\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.312987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-multus-conf-dir\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.313016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-slash\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.313055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-var-lib-openvswitch\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.313079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-run-multus-certs\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.313195 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-run-multus-certs\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.313268 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.313266 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/679a7244-b847-4ffc-b0c1-6c9c023eb440-env-overrides\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.313329 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15335f10-5396-417f-9436-0b8abd1664be-multus-daemon-config\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.313378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.313344 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-run-k8s-cni-cncf-io\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.321560 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.321532 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-cni-netd\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.322029 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.322006 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/679a7244-b847-4ffc-b0c1-6c9c023eb440-ovnkube-config\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.322113 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.322097 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-os-release\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.324520 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7d55ba4-f5a8-48fd-8c9f-932ceaedc498-agent-certs\") pod \"konnectivity-agent-n88bh\" (UID: \"c7d55ba4-f5a8-48fd-8c9f-932ceaedc498\") " pod="kube-system/konnectivity-agent-n88bh" Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.324561 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7d55ba4-f5a8-48fd-8c9f-932ceaedc498-konnectivity-ca\") pod \"konnectivity-agent-n88bh\" (UID: \"c7d55ba4-f5a8-48fd-8c9f-932ceaedc498\") " pod="kube-system/konnectivity-agent-n88bh" Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.324641 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-multus-conf-dir\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.324741 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f2af6702-608a-497a-bcc5-d7eac4802553-serviceca\") pod \"node-ca-vthpv\" (UID: \"f2af6702-608a-497a-bcc5-d7eac4802553\") " pod="openshift-image-registry/node-ca-vthpv" Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.324785 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.324899 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-systemd-units\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.324964 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-multus-socket-dir-parent\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.324990 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret podName:bbf8d227-a317-4952-a0b2-7ca6d602d804 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:17.82495162 +0000 UTC m=+3.084242358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret") pod "global-pull-secret-syncer-cg9km" (UID: "bbf8d227-a317-4952-a0b2-7ca6d602d804") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.325001 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-var-lib-cni-bin\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.325040 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.325049 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-host-run-netns\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.325055 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/679a7244-b847-4ffc-b0c1-6c9c023eb440-ovn-node-metrics-cert\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.325069 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-var-lib-openvswitch\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.325079 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.325078 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15335f10-5396-417f-9436-0b8abd1664be-cni-binary-copy\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.325089 2573 projected.go:196] Error preparing data for projected volume kube-api-access-g2ldm for pod openshift-network-diagnostics/network-check-target-564mm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.325043 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-slash\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.326329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.325126 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbf8d227-a317-4952-a0b2-7ca6d602d804-kubelet-config\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:17.327172 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.325284 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15335f10-5396-417f-9436-0b8abd1664be-hostroot\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.327172 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.325310 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm podName:42398dea-bac1-4abf-accc-beca393738af nodeName:}" failed. No retries permitted until 2026-07-28 12:30:17.82529289 +0000 UTC m=+3.084583608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g2ldm" (UniqueName: "kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm") pod "network-check-target-564mm" (UID: "42398dea-bac1-4abf-accc-beca393738af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:17.327172 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.325333 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-host-run-netns\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.327172 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.325351 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-node-log\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.327172 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.325349 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-etc-openvswitch\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.327172 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.325384 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/679a7244-b847-4ffc-b0c1-6c9c023eb440-run-ovn\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.327172 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.325399 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:17.327172 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.325510 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs podName:a4868cb1-759e-4f1a-a614-165690edf4a3 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:17.825485436 +0000 UTC m=+3.084776178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs") pod "network-metrics-daemon-csfdh" (UID: "a4868cb1-759e-4f1a-a614-165690edf4a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:17.327783 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.327756 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9xh\" (UniqueName: \"kubernetes.io/projected/679a7244-b847-4ffc-b0c1-6c9c023eb440-kube-api-access-wp9xh\") pod \"ovnkube-node-kfk7p\" (UID: \"679a7244-b847-4ffc-b0c1-6c9c023eb440\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.328670 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.328638 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bdfw\" (UniqueName: \"kubernetes.io/projected/a4868cb1-759e-4f1a-a614-165690edf4a3-kube-api-access-7bdfw\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:17.329386 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.329299 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvdp\" (UniqueName: \"kubernetes.io/projected/15335f10-5396-417f-9436-0b8abd1664be-kube-api-access-dcvdp\") pod \"multus-w262h\" (UID: \"15335f10-5396-417f-9436-0b8abd1664be\") " pod="openshift-multus/multus-w262h" Jul 28 12:30:17.329721 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.329702 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmwhj\" (UniqueName: \"kubernetes.io/projected/f2af6702-608a-497a-bcc5-d7eac4802553-kube-api-access-pmwhj\") pod \"node-ca-vthpv\" (UID: \"f2af6702-608a-497a-bcc5-d7eac4802553\") " pod="openshift-image-registry/node-ca-vthpv" Jul 28 12:30:17.410991 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.410957 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-strvp" Jul 28 12:30:17.422895 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.422846 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-54zx6" Jul 28 12:30:17.434616 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.434588 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" Jul 28 12:30:17.439867 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.439846 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" Jul 28 12:30:17.448422 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.448403 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wd7wp" Jul 28 12:30:17.458200 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.458184 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:17.464843 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.464827 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n88bh" Jul 28 12:30:17.472505 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.472485 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w262h" Jul 28 12:30:17.479241 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.479216 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vthpv" Jul 28 12:30:17.918160 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.918076 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ldm\" (UniqueName: \"kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm\") pod \"network-check-target-564mm\" (UID: \"42398dea-bac1-4abf-accc-beca393738af\") " pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:17.918160 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.918134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:17.918384 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:17.918173 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:17.918384 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.918277 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:17.918384 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.918300 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:30:17.918384 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.918324 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:30:17.918384 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.918337 2573 projected.go:196] Error preparing data for projected volume kube-api-access-g2ldm for pod openshift-network-diagnostics/network-check-target-564mm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:17.918384 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.918359 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs podName:a4868cb1-759e-4f1a-a614-165690edf4a3 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:18.918337968 +0000 UTC m=+4.177628689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs") pod "network-metrics-daemon-csfdh" (UID: "a4868cb1-759e-4f1a-a614-165690edf4a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:17.918384 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.918277 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:17.918700 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.918389 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm podName:42398dea-bac1-4abf-accc-beca393738af nodeName:}" failed. No retries permitted until 2026-07-28 12:30:18.918372209 +0000 UTC m=+4.177662941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2ldm" (UniqueName: "kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm") pod "network-check-target-564mm" (UID: "42398dea-bac1-4abf-accc-beca393738af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:17.918700 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:17.918417 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret podName:bbf8d227-a317-4952-a0b2-7ca6d602d804 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:18.918402439 +0000 UTC m=+4.177693162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret") pod "global-pull-secret-syncer-cg9km" (UID: "bbf8d227-a317-4952-a0b2-7ca6d602d804") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:18.118639 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:18.118612 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 12:30:18.261705 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:18.261674 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:18.261864 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:18.261778 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:18.269986 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:18.269962 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:25:17 +0000 UTC" deadline="2028-01-30 20:51:38.189479317 +0000 UTC" Jul 28 12:30:18.269986 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:18.269984 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13232h21m19.919499768s" Jul 28 12:30:18.326718 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:18.326690 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fba58b4_2405_4d1c_8413_219e279f973a.slice/crio-741bebfd026f49febfcc3b4d3d7568f8400ba2d4897f353b7b5f1459007fb703 WatchSource:0}: Error finding container 741bebfd026f49febfcc3b4d3d7568f8400ba2d4897f353b7b5f1459007fb703: Status 404 returned error can't find the container with id 741bebfd026f49febfcc3b4d3d7568f8400ba2d4897f353b7b5f1459007fb703 Jul 28 12:30:18.329597 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:18.329572 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2af6702_608a_497a_bcc5_d7eac4802553.slice/crio-0025d722475096c07346518f5702bc50791b5e029984d9cc55dc15b63c6f1d03 WatchSource:0}: Error finding container 0025d722475096c07346518f5702bc50791b5e029984d9cc55dc15b63c6f1d03: Status 404 returned error can't find the container with id 0025d722475096c07346518f5702bc50791b5e029984d9cc55dc15b63c6f1d03 Jul 28 12:30:18.333934 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:18.333913 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32eea5ab_06de_4cc0_ab3d_16c855389a86.slice/crio-516b101d812ca283359ff221c776f08a1139b0f3064c25de6174a464513e9619 WatchSource:0}: Error finding container 516b101d812ca283359ff221c776f08a1139b0f3064c25de6174a464513e9619: Status 404 returned error can't find the container with id 516b101d812ca283359ff221c776f08a1139b0f3064c25de6174a464513e9619 Jul 28 12:30:18.344734 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:18.344703 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51971a1f_fb2e_481f_8a79_b80660f239fd.slice/crio-fbdb79601441a626b7ea5ca21bb84984f7e93cbe99fdf2f0e08984b2e71e8747 WatchSource:0}: Error finding container fbdb79601441a626b7ea5ca21bb84984f7e93cbe99fdf2f0e08984b2e71e8747: Status 404 returned error can't find the container with id fbdb79601441a626b7ea5ca21bb84984f7e93cbe99fdf2f0e08984b2e71e8747 Jul 28 12:30:18.348128 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:18.347694 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d55ba4_f5a8_48fd_8c9f_932ceaedc498.slice/crio-1a7040afa47cbc39d9e84a9ad461d3a85fe8e96ccb1451ae7336b925008946c3 WatchSource:0}: Error finding container 1a7040afa47cbc39d9e84a9ad461d3a85fe8e96ccb1451ae7336b925008946c3: Status 404 returned error can't find the container with id 1a7040afa47cbc39d9e84a9ad461d3a85fe8e96ccb1451ae7336b925008946c3 Jul 28 12:30:18.348128 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:18.348063 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15335f10_5396_417f_9436_0b8abd1664be.slice/crio-e1bd9614b62834674243d2519b9fd5812a09dbadd0175beba66569a46a58e21f WatchSource:0}: Error finding container e1bd9614b62834674243d2519b9fd5812a09dbadd0175beba66569a46a58e21f: Status 404 returned error can't find the container with id e1bd9614b62834674243d2519b9fd5812a09dbadd0175beba66569a46a58e21f Jul 28 12:30:18.929406 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:18.929083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ldm\" (UniqueName: \"kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm\") pod \"network-check-target-564mm\" (UID: \"42398dea-bac1-4abf-accc-beca393738af\") " pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:18.929406 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:18.929350 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:18.929406 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:18.929289 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:30:18.929406 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:18.929396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:18.929406 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:18.929406 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:30:18.929801 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:18.929429 2573 projected.go:196] Error preparing data for projected volume kube-api-access-g2ldm for pod openshift-network-diagnostics/network-check-target-564mm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:18.929801 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:18.929511 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm podName:42398dea-bac1-4abf-accc-beca393738af nodeName:}" failed. No retries permitted until 2026-07-28 12:30:20.929492623 +0000 UTC m=+6.188783357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2ldm" (UniqueName: "kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm") pod "network-check-target-564mm" (UID: "42398dea-bac1-4abf-accc-beca393738af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:18.929801 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:18.929549 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:18.929801 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:18.929586 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret podName:bbf8d227-a317-4952-a0b2-7ca6d602d804 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:20.929575681 +0000 UTC m=+6.188866403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret") pod "global-pull-secret-syncer-cg9km" (UID: "bbf8d227-a317-4952-a0b2-7ca6d602d804") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:18.929801 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:18.929655 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:18.929801 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:18.929684 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs podName:a4868cb1-759e-4f1a-a614-165690edf4a3 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:20.929675157 +0000 UTC m=+6.188965877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs") pod "network-metrics-daemon-csfdh" (UID: "a4868cb1-759e-4f1a-a614-165690edf4a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:19.264146 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.263572 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:19.264146 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.263576 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:19.264146 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:19.263723 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:19.264146 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:19.263869 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:19.270226 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.270111 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:25:17 +0000 UTC" deadline="2028-02-17 08:55:19.176986838 +0000 UTC" Jul 28 12:30:19.270226 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.270197 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13652h24m59.906795457s" Jul 28 12:30:19.292834 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.292803 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wd7wp" event={"ID":"8325bc39-4cb3-49bb-b803-18c2ab4a24e7","Type":"ContainerStarted","Data":"5d3a8fd4f32c5df2be4f753d89abf6df82a4735f5a3f0e87a39e36651ec7f4a1"} Jul 28 12:30:19.307124 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.307049 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w262h" event={"ID":"15335f10-5396-417f-9436-0b8abd1664be","Type":"ContainerStarted","Data":"e1bd9614b62834674243d2519b9fd5812a09dbadd0175beba66569a46a58e21f"} Jul 28 12:30:19.312352 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.312257 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" event={"ID":"51971a1f-fb2e-481f-8a79-b80660f239fd","Type":"ContainerStarted","Data":"fbdb79601441a626b7ea5ca21bb84984f7e93cbe99fdf2f0e08984b2e71e8747"} Jul 28 12:30:19.331202 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.331175 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-145.ec2.internal" event={"ID":"4541c39d7375e0d14063052c99c9a71f","Type":"ContainerStarted","Data":"2a3b36aec5667f87460fe1fcbcb53bb151cf11952ccd6cd57063628dea6b322b"} Jul 28 12:30:19.348750 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.348716 2573 generic.go:356] "Generic (PLEG): container finished" podID="62537a11327111f64b516ac280dab601" containerID="c05025fa1c86738d835705b7f9ef536389545f5e757c6d9071808fb2bf8ebade" exitCode=0 Jul 28 12:30:19.348862 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.348795 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal" event={"ID":"62537a11327111f64b516ac280dab601","Type":"ContainerDied","Data":"c05025fa1c86738d835705b7f9ef536389545f5e757c6d9071808fb2bf8ebade"} Jul 28 12:30:19.371313 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.366892 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" event={"ID":"16de775e-beb3-4834-9a10-724db2f7003b","Type":"ContainerStarted","Data":"6d563d06b5de94ea542f57406a7a5746829556168ee7f34bd9d0c22c5b7fe8f5"} Jul 28 12:30:19.371313 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.368351 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-145.ec2.internal" podStartSLOduration=3.368333544 podStartE2EDuration="3.368333544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:19.348188655 +0000 UTC m=+4.607479397" watchObservedRunningTime="2026-07-28 12:30:19.368333544 +0000 UTC m=+4.627624287" Jul 28 12:30:19.371313 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.371272 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n88bh" event={"ID":"c7d55ba4-f5a8-48fd-8c9f-932ceaedc498","Type":"ContainerStarted","Data":"1a7040afa47cbc39d9e84a9ad461d3a85fe8e96ccb1451ae7336b925008946c3"} Jul 28 12:30:19.393410 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.393379 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" event={"ID":"679a7244-b847-4ffc-b0c1-6c9c023eb440","Type":"ContainerStarted","Data":"a2b9a57f0e9eb0f9958d3c1fe3f8ecbf27177c339077862489e226cc59977abf"} Jul 28 12:30:19.399847 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.399819 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-strvp" event={"ID":"32eea5ab-06de-4cc0-ab3d-16c855389a86","Type":"ContainerStarted","Data":"516b101d812ca283359ff221c776f08a1139b0f3064c25de6174a464513e9619"} Jul 28 12:30:19.416560 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.416494 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vthpv" event={"ID":"f2af6702-608a-497a-bcc5-d7eac4802553","Type":"ContainerStarted","Data":"0025d722475096c07346518f5702bc50791b5e029984d9cc55dc15b63c6f1d03"} Jul 28 12:30:19.429143 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:19.429116 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-54zx6" event={"ID":"9fba58b4-2405-4d1c-8413-219e279f973a","Type":"ContainerStarted","Data":"741bebfd026f49febfcc3b4d3d7568f8400ba2d4897f353b7b5f1459007fb703"} Jul 28 12:30:20.262166 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:20.262136 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:20.262363 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:20.262263 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:20.451378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:20.451342 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal" event={"ID":"62537a11327111f64b516ac280dab601","Type":"ContainerStarted","Data":"0c2049bd9023c4376b3d6dfec0ffc8bc61f5537e29656514bb9b405f0561d3a2"} Jul 28 12:30:20.949653 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:20.949612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ldm\" (UniqueName: \"kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm\") pod \"network-check-target-564mm\" (UID: \"42398dea-bac1-4abf-accc-beca393738af\") " pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:20.949840 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:20.949722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:20.949840 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:20.949759 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:20.949962 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:20.949880 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:20.949962 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:20.949941 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret podName:bbf8d227-a317-4952-a0b2-7ca6d602d804 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:24.949923675 +0000 UTC m=+10.209214397 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret") pod "global-pull-secret-syncer-cg9km" (UID: "bbf8d227-a317-4952-a0b2-7ca6d602d804") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:20.950775 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:20.950409 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:30:20.950775 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:20.950433 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:30:20.950775 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:20.950465 2573 projected.go:196] Error preparing data for projected volume kube-api-access-g2ldm for pod openshift-network-diagnostics/network-check-target-564mm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:20.950775 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:20.950538 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm podName:42398dea-bac1-4abf-accc-beca393738af nodeName:}" failed. No retries permitted until 2026-07-28 12:30:24.950522104 +0000 UTC m=+10.209812825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2ldm" (UniqueName: "kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm") pod "network-check-target-564mm" (UID: "42398dea-bac1-4abf-accc-beca393738af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:20.950775 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:20.950606 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:20.950775 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:20.950639 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs podName:a4868cb1-759e-4f1a-a614-165690edf4a3 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:24.950626796 +0000 UTC m=+10.209917516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs") pod "network-metrics-daemon-csfdh" (UID: "a4868cb1-759e-4f1a-a614-165690edf4a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:21.264458 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:21.264364 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:21.264616 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:21.264525 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:21.264926 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:21.264904 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:21.265021 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:21.264996 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:22.262479 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:22.261953 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:22.262479 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:22.262096 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:23.263578 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:23.263546 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:23.263578 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:23.263563 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:23.264212 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:23.263675 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:23.264212 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:23.263810 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:24.262808 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:24.262311 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:24.262808 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:24.262466 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:24.984597 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:24.984548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ldm\" (UniqueName: \"kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm\") pod \"network-check-target-564mm\" (UID: \"42398dea-bac1-4abf-accc-beca393738af\") " pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:24.985058 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:24.984616 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:24.985058 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:24.984641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:24.985058 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:24.984757 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:24.985058 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:24.984805 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret podName:bbf8d227-a317-4952-a0b2-7ca6d602d804 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:32.984793533 +0000 UTC m=+18.244084265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret") pod "global-pull-secret-syncer-cg9km" (UID: "bbf8d227-a317-4952-a0b2-7ca6d602d804") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:24.985272 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:24.985157 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:30:24.985272 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:24.985170 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:30:24.985272 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:24.985178 2573 projected.go:196] Error preparing data for projected volume kube-api-access-g2ldm for pod openshift-network-diagnostics/network-check-target-564mm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:24.985272 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:24.985210 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm podName:42398dea-bac1-4abf-accc-beca393738af nodeName:}" failed. No retries permitted until 2026-07-28 12:30:32.985200356 +0000 UTC m=+18.244491089 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2ldm" (UniqueName: "kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm") pod "network-check-target-564mm" (UID: "42398dea-bac1-4abf-accc-beca393738af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:24.985272 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:24.985267 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:24.985522 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:24.985302 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs podName:a4868cb1-759e-4f1a-a614-165690edf4a3 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:32.985289192 +0000 UTC m=+18.244579928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs") pod "network-metrics-daemon-csfdh" (UID: "a4868cb1-759e-4f1a-a614-165690edf4a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:25.263649 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:25.263571 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:25.263805 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:25.263713 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:25.264075 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:25.264053 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:25.264168 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:25.264145 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:26.262177 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:26.262143 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:26.262682 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:26.262283 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:27.262100 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:27.262065 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:27.262252 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:27.262226 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:27.262681 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:27.262579 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:27.262681 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:27.262648 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:28.262269 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:28.262239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:28.266002 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:28.262344 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:29.261830 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:29.261786 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:29.261958 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:29.261937 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:29.262015 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:29.261992 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:29.262113 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:29.262093 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:30.262203 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:30.262164 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:30.262684 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:30.262306 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:31.261851 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:31.261812 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:31.262022 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:31.261859 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:31.262022 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:31.261977 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:31.262130 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:31.262089 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:32.262367 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:32.262328 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:32.262875 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:32.262470 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:33.047674 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:33.047641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ldm\" (UniqueName: \"kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm\") pod \"network-check-target-564mm\" (UID: \"42398dea-bac1-4abf-accc-beca393738af\") " pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:33.047674 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:33.047689 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:33.047901 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:33.047719 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:33.047901 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:33.047811 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:33.047901 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:33.047813 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:33.047901 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:33.047865 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret podName:bbf8d227-a317-4952-a0b2-7ca6d602d804 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:49.047850919 +0000 UTC m=+34.307141657 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret") pod "global-pull-secret-syncer-cg9km" (UID: "bbf8d227-a317-4952-a0b2-7ca6d602d804") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:33.047901 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:33.047895 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs podName:a4868cb1-759e-4f1a-a614-165690edf4a3 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:49.047873926 +0000 UTC m=+34.307164691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs") pod "network-metrics-daemon-csfdh" (UID: "a4868cb1-759e-4f1a-a614-165690edf4a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:33.047901 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:33.047815 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:30:33.048135 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:33.047920 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:30:33.048135 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:33.047930 2573 projected.go:196] Error preparing data for projected volume kube-api-access-g2ldm for pod openshift-network-diagnostics/network-check-target-564mm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:33.048135 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:33.047959 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm podName:42398dea-bac1-4abf-accc-beca393738af nodeName:}" failed. No retries permitted until 2026-07-28 12:30:49.047952171 +0000 UTC m=+34.307242889 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2ldm" (UniqueName: "kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm") pod "network-check-target-564mm" (UID: "42398dea-bac1-4abf-accc-beca393738af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:33.261997 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:33.261960 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:33.262164 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:33.261960 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:33.262164 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:33.262089 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:33.262164 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:33.262135 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:34.262333 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:34.262293 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:34.262789 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:34.262428 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:35.262873 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:35.262838 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:35.263336 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:35.262887 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:35.263336 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:35.262966 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:35.263336 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:35.263081 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:36.262501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:36.262470 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:36.262667 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:36.262582 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:37.263887 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.263862 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:37.264258 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:37.263995 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:37.264358 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.264339 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:37.264473 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:37.264430 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:37.485676 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.485267 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" event={"ID":"16de775e-beb3-4834-9a10-724db2f7003b","Type":"ContainerStarted","Data":"08d9c25d01aaa13ef3982b652239527a6ed4955357301b115e7518458c3ec401"} Jul 28 12:30:37.486902 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.486870 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n88bh" event={"ID":"c7d55ba4-f5a8-48fd-8c9f-932ceaedc498","Type":"ContainerStarted","Data":"83d91219af3af8586429527babe6e68fee9749b087431eff056320c88bc5382c"} Jul 28 12:30:37.489032 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.489006 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" event={"ID":"679a7244-b847-4ffc-b0c1-6c9c023eb440","Type":"ContainerStarted","Data":"810b2be9a86a8f6efff90fb153be12256e9edc7685d7249c56602d00173226ca"} Jul 28 12:30:37.489135 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.489076 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" event={"ID":"679a7244-b847-4ffc-b0c1-6c9c023eb440","Type":"ContainerStarted","Data":"7d222b46d56f34e2ee0bd56fbaa5bc11810662ed3494b3326c216f95b42104a9"} Jul 28 12:30:37.491178 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.490662 2573 generic.go:356] "Generic (PLEG): container finished" podID="32eea5ab-06de-4cc0-ab3d-16c855389a86" containerID="248b5021f2e2d0ef032bf828907ccb64728e32e3b12b31fd1088782f219bdac2" exitCode=0 Jul 28 12:30:37.491178 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.490742 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-strvp" event={"ID":"32eea5ab-06de-4cc0-ab3d-16c855389a86","Type":"ContainerDied","Data":"248b5021f2e2d0ef032bf828907ccb64728e32e3b12b31fd1088782f219bdac2"} Jul 28 12:30:37.493477 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.492280 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vthpv" event={"ID":"f2af6702-608a-497a-bcc5-d7eac4802553","Type":"ContainerStarted","Data":"ad93003ed9fec4c4b0574c7066ba600ce1886dfae98ce84a414a14af493bb3f6"} Jul 28 12:30:37.495059 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.495039 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wd7wp" event={"ID":"8325bc39-4cb3-49bb-b803-18c2ab4a24e7","Type":"ContainerStarted","Data":"0cbb7ab48a7e72cc4a8c4a902dd24a8a2c61affb404727e070cb9229f130594d"} Jul 28 12:30:37.496658 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.496636 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w262h" event={"ID":"15335f10-5396-417f-9436-0b8abd1664be","Type":"ContainerStarted","Data":"0a0fe536f72d183bbbd7b6c075bd6c5ded72ac558164cd4afd82d389ece3a660"} Jul 28 12:30:37.498112 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.498088 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" event={"ID":"51971a1f-fb2e-481f-8a79-b80660f239fd","Type":"ContainerStarted","Data":"8184e59b5ef27e8f6fd725ba826fc432f163dea97ebc3cf23085ee3e74fddcd7"} Jul 28 12:30:37.504606 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.504569 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5jzt2" podStartSLOduration=3.67772606 podStartE2EDuration="22.504558199s" totalImagesPullingTime="18.826832139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:37.50453212 +0000 UTC m=+22.763822860" watchObservedRunningTime="2026-07-28 12:30:37.504558199 +0000 UTC m=+22.763848940" Jul 28 12:30:37.504749 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.504725 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-145.ec2.internal" podStartSLOduration=21.504720671 podStartE2EDuration="21.504720671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:20.47184895 +0000 UTC m=+5.731139692" watchObservedRunningTime="2026-07-28 12:30:37.504720671 +0000 UTC m=+22.764011412" Jul 28 12:30:37.521528 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.521493 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n88bh" podStartSLOduration=7.5522562319999995 podStartE2EDuration="22.521476213s" totalImagesPullingTime="14.969219981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:37.520939635 +0000 UTC m=+22.780230376" watchObservedRunningTime="2026-07-28 12:30:37.521476213 +0000 UTC m=+22.780766955" Jul 28 12:30:37.539021 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.538967 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w262h" podStartSLOduration=3.6656037550000002 podStartE2EDuration="22.538955991s" totalImagesPullingTime="18.873352236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:37.538882363 +0000 UTC m=+22.798173121" watchObservedRunningTime="2026-07-28 12:30:37.538955991 +0000 UTC m=+22.798246736" Jul 28 12:30:37.555327 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.555284 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vthpv" podStartSLOduration=3.70997924 podStartE2EDuration="22.555271842s" totalImagesPullingTime="18.845292602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:37.555012114 +0000 UTC m=+22.814302855" watchObservedRunningTime="2026-07-28 12:30:37.555271842 +0000 UTC m=+22.814562582" Jul 28 12:30:37.598111 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:37.598070 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wd7wp" podStartSLOduration=3.794652429 podStartE2EDuration="22.598056251s" totalImagesPullingTime="18.803403822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:37.597772187 +0000 UTC m=+22.857062951" watchObservedRunningTime="2026-07-28 12:30:37.598056251 +0000 UTC m=+22.857346991" Jul 28 12:30:38.262111 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:38.262043 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:38.262259 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:38.262215 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:38.504005 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:38.503972 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" event={"ID":"679a7244-b847-4ffc-b0c1-6c9c023eb440","Type":"ContainerStarted","Data":"48b939301a6a2aaba735bb334c9a9087d0c79499a4a3098f8f62429de271d1da"} Jul 28 12:30:38.504734 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:38.504050 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" event={"ID":"679a7244-b847-4ffc-b0c1-6c9c023eb440","Type":"ContainerStarted","Data":"95475fa458cf4c857c584f2f2a9ef9fd3b39cb6e6c4a47488cbca762afe479c4"} Jul 28 12:30:38.504734 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:38.504082 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" event={"ID":"679a7244-b847-4ffc-b0c1-6c9c023eb440","Type":"ContainerStarted","Data":"ac60b5c83feda4a1fd544593e9eae89770b2a7d52ebef7aab6ff067a656ad585"} Jul 28 12:30:38.504734 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:38.504118 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" event={"ID":"679a7244-b847-4ffc-b0c1-6c9c023eb440","Type":"ContainerStarted","Data":"3fe21719efd1d5390716d44ad97d04002edad9fb70eb7a38f4febc8f0f672f77"} Jul 28 12:30:38.656545 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:38.656517 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 12:30:39.172827 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:39.172711 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T12:30:38.656539148Z","UUID":"becf32df-ba74-41a7-8582-d3371a8e3234","Handler":null,"Name":"","Endpoint":""} Jul 28 12:30:39.174737 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:39.174705 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 12:30:39.174877 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:39.174748 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 12:30:39.262282 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:39.262250 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:39.262489 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:39.262385 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:39.262489 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:39.262456 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:39.262616 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:39.262534 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:39.508861 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:39.508773 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" event={"ID":"51971a1f-fb2e-481f-8a79-b80660f239fd","Type":"ContainerStarted","Data":"a5e3ce135cf0e357c5458d6f74032ff09112b7ee685e5a7f8aa7ff8bf649860b"} Jul 28 12:30:39.511646 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:39.511614 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-54zx6" event={"ID":"9fba58b4-2405-4d1c-8413-219e279f973a","Type":"ContainerStarted","Data":"8a6356f136109dd5c0f77aa11fbba5baf15c5157648aa4b9c65313856e651448"} Jul 28 12:30:39.529098 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:39.529057 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-54zx6" podStartSLOduration=5.680603229 podStartE2EDuration="24.529044683s" totalImagesPullingTime="18.848441454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:39.528776922 +0000 UTC m=+24.788067666" watchObservedRunningTime="2026-07-28 12:30:39.529044683 +0000 UTC m=+24.788335424" Jul 28 12:30:40.261745 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:40.261720 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:40.261886 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:40.261813 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:40.517785 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:40.517549 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" event={"ID":"679a7244-b847-4ffc-b0c1-6c9c023eb440","Type":"ContainerStarted","Data":"50ce0a6e593894829f9fdb3562728cf6910f7ba2d31afbdeba681734d0e765f4"} Jul 28 12:30:40.519894 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:40.519865 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" event={"ID":"51971a1f-fb2e-481f-8a79-b80660f239fd","Type":"ContainerStarted","Data":"88f0b338ea85543c281ddd83e2c059d7a12710ee997aabfab224e4891434d96b"} Jul 28 12:30:40.539860 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:40.539809 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-69gmf" podStartSLOduration=4.296534638 podStartE2EDuration="25.539791825s" totalImagesPullingTime="21.243257187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:30:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:40.539422679 +0000 UTC m=+25.798713420" watchObservedRunningTime="2026-07-28 12:30:40.539791825 +0000 UTC m=+25.799082563" Jul 28 12:30:41.262084 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:41.262052 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:41.262252 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:41.262053 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:41.262252 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:41.262165 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:41.262357 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:41.262269 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:41.925427 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:41.925394 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n88bh" Jul 28 12:30:41.925977 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:41.925958 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n88bh" Jul 28 12:30:42.262096 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:42.262069 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:42.262232 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:42.262208 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:42.528265 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:42.527960 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" event={"ID":"679a7244-b847-4ffc-b0c1-6c9c023eb440","Type":"ContainerStarted","Data":"3ffef1649d52b9fa3357ab2d9e4b761095abb705b979bb3e6910aaa7819730b4"} Jul 28 12:30:42.528265 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:42.528255 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:42.530029 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:42.530002 2573 generic.go:356] "Generic (PLEG): container finished" podID="32eea5ab-06de-4cc0-ab3d-16c855389a86" containerID="abaaede8e6078a708a032dc25eace630faa44c2f05d4d2ac7272e10f500349df" exitCode=0 Jul 28 12:30:42.530189 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:42.530084 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-strvp" event={"ID":"32eea5ab-06de-4cc0-ab3d-16c855389a86","Type":"ContainerDied","Data":"abaaede8e6078a708a032dc25eace630faa44c2f05d4d2ac7272e10f500349df"} Jul 28 12:30:42.530302 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:42.530285 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n88bh" Jul 28 12:30:42.530993 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:42.530971 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n88bh" Jul 28 12:30:42.545508 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:42.545489 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:42.556770 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:42.556730 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" podStartSLOduration=8.659406929 podStartE2EDuration="27.556715877s" totalImagesPullingTime="18.897308948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:42.556201851 +0000 UTC m=+27.815492592" watchObservedRunningTime="2026-07-28 12:30:42.556715877 +0000 UTC m=+27.816006618" Jul 28 12:30:43.262500 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:43.262413 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:43.263428 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:43.262428 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:43.263428 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:43.262578 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:43.263428 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:43.262601 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:43.532889 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:43.532823 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:43.532889 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:43.532880 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:43.555023 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:43.554991 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:30:44.261825 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:44.261795 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:44.262018 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:44.261928 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:44.326056 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:44.326018 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-564mm"] Jul 28 12:30:44.326589 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:44.326139 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:44.326589 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:44.326240 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:44.326710 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:44.326691 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-csfdh"] Jul 28 12:30:44.326824 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:44.326808 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:44.326933 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:44.326912 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:44.327290 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:44.327270 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cg9km"] Jul 28 12:30:44.536670 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:44.536579 2573 generic.go:356] "Generic (PLEG): container finished" podID="32eea5ab-06de-4cc0-ab3d-16c855389a86" containerID="cb285a31a1db06d8ad9ed37c409c781891ef7bd0bd7f5d8f618bf001431367ea" exitCode=0 Jul 28 12:30:44.536670 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:44.536658 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:44.536923 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:44.536680 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-strvp" event={"ID":"32eea5ab-06de-4cc0-ab3d-16c855389a86","Type":"ContainerDied","Data":"cb285a31a1db06d8ad9ed37c409c781891ef7bd0bd7f5d8f618bf001431367ea"} Jul 28 12:30:44.536923 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:44.536761 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:46.262300 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:46.262087 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:46.262774 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:46.262092 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:46.262774 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:46.262316 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:46.262774 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:46.262128 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:46.262774 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:46.262415 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:46.262774 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:46.262555 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:46.542499 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:46.542406 2573 generic.go:356] "Generic (PLEG): container finished" podID="32eea5ab-06de-4cc0-ab3d-16c855389a86" containerID="73e2024327746200b8b999ade50a757dfe53768815539d00513d51bebd88346f" exitCode=0 Jul 28 12:30:46.542499 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:46.542476 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-strvp" event={"ID":"32eea5ab-06de-4cc0-ab3d-16c855389a86","Type":"ContainerDied","Data":"73e2024327746200b8b999ade50a757dfe53768815539d00513d51bebd88346f"} Jul 28 12:30:48.261825 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:48.261794 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:48.262513 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:48.261848 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:48.262513 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:48.261858 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:48.262513 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:48.262021 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg9km" podUID="bbf8d227-a317-4952-a0b2-7ca6d602d804" Jul 28 12:30:48.262513 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:48.262044 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:30:48.262513 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:48.262079 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-564mm" podUID="42398dea-bac1-4abf-accc-beca393738af" Jul 28 12:30:49.065786 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:49.065753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ldm\" (UniqueName: \"kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm\") pod \"network-check-target-564mm\" (UID: \"42398dea-bac1-4abf-accc-beca393738af\") " pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:49.065786 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:49.065801 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:49.066032 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:49.065836 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:49.066032 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:49.065955 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:49.066032 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:49.065956 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:30:49.066032 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:49.065988 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:30:49.066032 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:49.066001 2573 projected.go:196] Error preparing data for projected volume kube-api-access-g2ldm for pod openshift-network-diagnostics/network-check-target-564mm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:49.066032 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:49.066013 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret podName:bbf8d227-a317-4952-a0b2-7ca6d602d804 nodeName:}" failed. No retries permitted until 2026-07-28 12:31:21.065999045 +0000 UTC m=+66.325289776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret") pod "global-pull-secret-syncer-cg9km" (UID: "bbf8d227-a317-4952-a0b2-7ca6d602d804") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:30:49.066032 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:49.066055 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm podName:42398dea-bac1-4abf-accc-beca393738af nodeName:}" failed. No retries permitted until 2026-07-28 12:31:21.066036871 +0000 UTC m=+66.325327592 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2ldm" (UniqueName: "kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm") pod "network-check-target-564mm" (UID: "42398dea-bac1-4abf-accc-beca393738af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:30:49.066452 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:49.066124 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:49.066452 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:49.066157 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs podName:a4868cb1-759e-4f1a-a614-165690edf4a3 nodeName:}" failed. No retries permitted until 2026-07-28 12:31:21.066145896 +0000 UTC m=+66.325436620 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs") pod "network-metrics-daemon-csfdh" (UID: "a4868cb1-759e-4f1a-a614-165690edf4a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:30:50.094934 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.094860 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-145.ec2.internal" event="NodeReady" Jul 28 12:30:50.095359 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.095006 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 12:30:50.140320 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.140285 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mfjqx"] Jul 28 12:30:50.142270 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.142244 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:50.143262 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.143242 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qfsc7"] Jul 28 12:30:50.145039 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.145022 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qfsc7" Jul 28 12:30:50.145292 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.145270 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 12:30:50.145384 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.145342 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bstgx\"" Jul 28 12:30:50.145645 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.145628 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 12:30:50.147663 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.147645 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-74b58\"" Jul 28 12:30:50.147903 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.147876 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 12:30:50.148279 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.148260 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 12:30:50.148706 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.148689 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 12:30:50.152628 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.152599 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mfjqx"] Jul 28 12:30:50.156872 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.156852 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qfsc7"] Jul 28 12:30:50.261642 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.261616 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:30:50.261815 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.261698 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:30:50.261876 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.261846 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:30:50.264981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.264753 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 12:30:50.264981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.264774 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 12:30:50.264981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.264786 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-klsqn\"" Jul 28 12:30:50.264981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.264835 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 12:30:50.264981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.264760 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nm4t9\"" Jul 28 12:30:50.265295 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.265106 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 12:30:50.272680 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.272662 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psjvl\" (UniqueName: \"kubernetes.io/projected/4a764e09-ea01-4cff-a9b1-0b14c995b676-kube-api-access-psjvl\") pod \"ingress-canary-qfsc7\" (UID: \"4a764e09-ea01-4cff-a9b1-0b14c995b676\") " pod="openshift-ingress-canary/ingress-canary-qfsc7" Jul 28 12:30:50.272786 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.272695 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-tmp-dir\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:50.272786 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.272744 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a764e09-ea01-4cff-a9b1-0b14c995b676-cert\") pod \"ingress-canary-qfsc7\" (UID: \"4a764e09-ea01-4cff-a9b1-0b14c995b676\") " pod="openshift-ingress-canary/ingress-canary-qfsc7" Jul 28 12:30:50.272786 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.272775 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qswdm\" (UniqueName: \"kubernetes.io/projected/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-kube-api-access-qswdm\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:50.272943 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.272804 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-config-volume\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:50.272943 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.272838 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:50.373252 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.373173 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a764e09-ea01-4cff-a9b1-0b14c995b676-cert\") pod \"ingress-canary-qfsc7\" (UID: \"4a764e09-ea01-4cff-a9b1-0b14c995b676\") " pod="openshift-ingress-canary/ingress-canary-qfsc7" Jul 28 12:30:50.373252 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.373221 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qswdm\" (UniqueName: \"kubernetes.io/projected/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-kube-api-access-qswdm\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:50.373483 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.373254 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-config-volume\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:50.373483 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.373429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:50.373583 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.373515 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psjvl\" (UniqueName: \"kubernetes.io/projected/4a764e09-ea01-4cff-a9b1-0b14c995b676-kube-api-access-psjvl\") pod \"ingress-canary-qfsc7\" (UID: \"4a764e09-ea01-4cff-a9b1-0b14c995b676\") " pod="openshift-ingress-canary/ingress-canary-qfsc7" Jul 28 12:30:50.373583 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:50.373551 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 12:30:50.373583 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.373556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-tmp-dir\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:50.373700 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:50.373636 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls podName:bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:50.873604722 +0000 UTC m=+36.132895457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls") pod "dns-default-mfjqx" (UID: "bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4") : secret "dns-default-metrics-tls" not found Jul 28 12:30:50.373887 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.373863 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-tmp-dir\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:50.373998 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.373915 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-config-volume\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:50.378119 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.378092 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a764e09-ea01-4cff-a9b1-0b14c995b676-cert\") pod \"ingress-canary-qfsc7\" (UID: \"4a764e09-ea01-4cff-a9b1-0b14c995b676\") " pod="openshift-ingress-canary/ingress-canary-qfsc7" Jul 28 12:30:50.382651 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.382629 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qswdm\" (UniqueName: \"kubernetes.io/projected/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-kube-api-access-qswdm\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:50.383320 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.383300 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psjvl\" (UniqueName: \"kubernetes.io/projected/4a764e09-ea01-4cff-a9b1-0b14c995b676-kube-api-access-psjvl\") pod \"ingress-canary-qfsc7\" (UID: \"4a764e09-ea01-4cff-a9b1-0b14c995b676\") " pod="openshift-ingress-canary/ingress-canary-qfsc7" Jul 28 12:30:50.463191 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.463159 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qfsc7" Jul 28 12:30:50.877001 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:50.876966 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:50.877181 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:50.877098 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 12:30:50.877181 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:50.877157 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls podName:bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:51.877141103 +0000 UTC m=+37.136431823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls") pod "dns-default-mfjqx" (UID: "bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4") : secret "dns-default-metrics-tls" not found Jul 28 12:30:51.883939 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:51.883906 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:51.884568 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:51.884047 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 12:30:51.884568 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:51.884099 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls podName:bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:53.884084651 +0000 UTC m=+39.143375384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls") pod "dns-default-mfjqx" (UID: "bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4") : secret "dns-default-metrics-tls" not found Jul 28 12:30:52.111189 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:52.111044 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qfsc7"] Jul 28 12:30:52.114760 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:30:52.114739 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a764e09_ea01_4cff_a9b1_0b14c995b676.slice/crio-f9125aef0428b3fe63750ea664c8b0be50fa0318f1e719765721f6c8827daaa6 WatchSource:0}: Error finding container f9125aef0428b3fe63750ea664c8b0be50fa0318f1e719765721f6c8827daaa6: Status 404 returned error can't find the container with id f9125aef0428b3fe63750ea664c8b0be50fa0318f1e719765721f6c8827daaa6 Jul 28 12:30:52.564133 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:52.564053 2573 generic.go:356] "Generic (PLEG): container finished" podID="32eea5ab-06de-4cc0-ab3d-16c855389a86" containerID="1823b22195e4506fec82d021fbcc713ec3cfe4d2b4e7b8d0ceb00068a98ecd23" exitCode=0 Jul 28 12:30:52.564290 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:52.564141 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-strvp" event={"ID":"32eea5ab-06de-4cc0-ab3d-16c855389a86","Type":"ContainerDied","Data":"1823b22195e4506fec82d021fbcc713ec3cfe4d2b4e7b8d0ceb00068a98ecd23"} Jul 28 12:30:52.565513 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:52.565484 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qfsc7" event={"ID":"4a764e09-ea01-4cff-a9b1-0b14c995b676","Type":"ContainerStarted","Data":"f9125aef0428b3fe63750ea664c8b0be50fa0318f1e719765721f6c8827daaa6"} Jul 28 12:30:53.571725 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:53.571510 2573 generic.go:356] "Generic (PLEG): container finished" podID="32eea5ab-06de-4cc0-ab3d-16c855389a86" containerID="4333c7a9ef545621574c29db0d761f7b6020720b36921d5442df2188dd45c9d5" exitCode=0 Jul 28 12:30:53.571725 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:53.571607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-strvp" event={"ID":"32eea5ab-06de-4cc0-ab3d-16c855389a86","Type":"ContainerDied","Data":"4333c7a9ef545621574c29db0d761f7b6020720b36921d5442df2188dd45c9d5"} Jul 28 12:30:53.898807 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:53.898748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:53.898931 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:53.898912 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 12:30:53.898994 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:53.898982 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls podName:bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4 nodeName:}" failed. No retries permitted until 2026-07-28 12:30:57.898966463 +0000 UTC m=+43.158257182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls") pod "dns-default-mfjqx" (UID: "bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4") : secret "dns-default-metrics-tls" not found Jul 28 12:30:54.575950 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:54.575918 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qfsc7" event={"ID":"4a764e09-ea01-4cff-a9b1-0b14c995b676","Type":"ContainerStarted","Data":"3e2fd4123a4b659b8ecde7c3cbcff95fa773f687b16d14a4beb1060045a0ea7d"} Jul 28 12:30:54.581936 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:54.581906 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-strvp" event={"ID":"32eea5ab-06de-4cc0-ab3d-16c855389a86","Type":"ContainerStarted","Data":"9682facf0ff0f52f93e039056382956e06648eca53944799cec2107c8fbd0246"} Jul 28 12:30:54.616640 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:54.616581 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qfsc7" podStartSLOduration=2.326927885 podStartE2EDuration="4.616559755s" totalImagesPullingTime="2.28963187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:54.592430182 +0000 UTC m=+39.851720923" watchObservedRunningTime="2026-07-28 12:30:54.616559755 +0000 UTC m=+39.875850498" Jul 28 12:30:54.618010 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:54.616956 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-strvp" podStartSLOduration=8.300542227 podStartE2EDuration="39.61694666s" totalImagesPullingTime="31.316404433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:30:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:54.614361464 +0000 UTC m=+39.873652229" watchObservedRunningTime="2026-07-28 12:30:54.61694666 +0000 UTC m=+39.876237405" Jul 28 12:30:57.926733 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:30:57.926689 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:30:57.927113 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:57.926839 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 12:30:57.927113 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:30:57.926900 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls podName:bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4 nodeName:}" failed. No retries permitted until 2026-07-28 12:31:05.926885323 +0000 UTC m=+51.186176042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls") pod "dns-default-mfjqx" (UID: "bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4") : secret "dns-default-metrics-tls" not found Jul 28 12:31:05.982459 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:05.982397 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:31:05.982910 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:31:05.982555 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 12:31:05.982910 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:31:05.982647 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls podName:bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4 nodeName:}" failed. No retries permitted until 2026-07-28 12:31:21.982630836 +0000 UTC m=+67.241921555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls") pod "dns-default-mfjqx" (UID: "bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4") : secret "dns-default-metrics-tls" not found Jul 28 12:31:15.549073 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:15.549046 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfk7p" Jul 28 12:31:21.083593 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.083550 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:31:21.084126 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.083618 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ldm\" (UniqueName: \"kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm\") pod \"network-check-target-564mm\" (UID: \"42398dea-bac1-4abf-accc-beca393738af\") " pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:31:21.084126 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.083663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:31:21.086987 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.086965 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 12:31:21.087075 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.087003 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 12:31:21.087075 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.087020 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 12:31:21.094805 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:31:21.094787 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 12:31:21.094870 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:31:21.094844 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs podName:a4868cb1-759e-4f1a-a614-165690edf4a3 nodeName:}" failed. No retries permitted until 2026-07-28 12:32:25.094827292 +0000 UTC m=+130.354118013 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs") pod "network-metrics-daemon-csfdh" (UID: "a4868cb1-759e-4f1a-a614-165690edf4a3") : secret "metrics-daemon-secret" not found Jul 28 12:31:21.097294 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.097272 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 12:31:21.098535 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.098517 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbf8d227-a317-4952-a0b2-7ca6d602d804-original-pull-secret\") pod \"global-pull-secret-syncer-cg9km\" (UID: \"bbf8d227-a317-4952-a0b2-7ca6d602d804\") " pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:31:21.107165 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.107141 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2ldm\" (UniqueName: \"kubernetes.io/projected/42398dea-bac1-4abf-accc-beca393738af-kube-api-access-g2ldm\") pod \"network-check-target-564mm\" (UID: \"42398dea-bac1-4abf-accc-beca393738af\") " pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:31:21.174977 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.174955 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-klsqn\"" Jul 28 12:31:21.178578 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.178565 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg9km" Jul 28 12:31:21.182145 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.182129 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:31:21.310709 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.310683 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-564mm"] Jul 28 12:31:21.313094 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:31:21.313063 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42398dea_bac1_4abf_accc_beca393738af.slice/crio-ab07ead7441c36e3f8cff302b333d517e25aec4842df8117a06ce2f715021a40 WatchSource:0}: Error finding container ab07ead7441c36e3f8cff302b333d517e25aec4842df8117a06ce2f715021a40: Status 404 returned error can't find the container with id ab07ead7441c36e3f8cff302b333d517e25aec4842df8117a06ce2f715021a40 Jul 28 12:31:21.321619 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.321599 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cg9km"] Jul 28 12:31:21.323772 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:31:21.323754 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf8d227_a317_4952_a0b2_7ca6d602d804.slice/crio-e136fa188b2892ce718380ac8796b36494dc8123419695f11d1535d14b327b7f WatchSource:0}: Error finding container e136fa188b2892ce718380ac8796b36494dc8123419695f11d1535d14b327b7f: Status 404 returned error can't find the container with id e136fa188b2892ce718380ac8796b36494dc8123419695f11d1535d14b327b7f Jul 28 12:31:21.644415 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.644378 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-564mm" event={"ID":"42398dea-bac1-4abf-accc-beca393738af","Type":"ContainerStarted","Data":"ab07ead7441c36e3f8cff302b333d517e25aec4842df8117a06ce2f715021a40"} Jul 28 12:31:21.645357 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.645330 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cg9km" event={"ID":"bbf8d227-a317-4952-a0b2-7ca6d602d804","Type":"ContainerStarted","Data":"e136fa188b2892ce718380ac8796b36494dc8123419695f11d1535d14b327b7f"} Jul 28 12:31:21.990675 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:21.990590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:31:21.990833 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:31:21.990751 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 12:31:21.990889 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:31:21.990839 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls podName:bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4 nodeName:}" failed. No retries permitted until 2026-07-28 12:31:53.990817796 +0000 UTC m=+99.250108517 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls") pod "dns-default-mfjqx" (UID: "bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4") : secret "dns-default-metrics-tls" not found Jul 28 12:31:26.661424 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:26.661384 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-564mm" event={"ID":"42398dea-bac1-4abf-accc-beca393738af","Type":"ContainerStarted","Data":"03152f64dd16cc7188da25d521572ae83048a8e4843e16bdb2bd6e161928a8e6"} Jul 28 12:31:26.661985 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:26.661508 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:31:26.662934 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:26.662913 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cg9km" event={"ID":"bbf8d227-a317-4952-a0b2-7ca6d602d804","Type":"ContainerStarted","Data":"6e632e59ce2c692cbfea89f2f082ea5aa9abb80c38bdae255e4fd2b8eda12246"} Jul 28 12:31:26.685074 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:26.685019 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-564mm" podStartSLOduration=67.010853182 podStartE2EDuration="1m11.685002499s" totalImagesPullingTime="4.674149317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:31:26.683704386 +0000 UTC m=+71.942995129" watchObservedRunningTime="2026-07-28 12:31:26.685002499 +0000 UTC m=+71.944293221" Jul 28 12:31:26.701936 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:26.701876 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cg9km" podStartSLOduration=67.033925418 podStartE2EDuration="1m11.701855945s" totalImagesPullingTime="4.667930527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:31:26.701481086 +0000 UTC m=+71.960771828" watchObservedRunningTime="2026-07-28 12:31:26.701855945 +0000 UTC m=+71.961146689" Jul 28 12:31:37.767991 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.767873 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw"] Jul 28 12:31:37.771098 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.771079 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z"] Jul 28 12:31:37.771214 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.771195 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:31:37.773996 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.773971 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z" Jul 28 12:31:37.775850 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.775829 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 28 12:31:37.775850 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.775844 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 28 12:31:37.776005 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.775867 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 28 12:31:37.776005 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.775948 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 28 12:31:37.777594 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.777576 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wfc89\"" Jul 28 12:31:37.777704 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.777615 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 28 12:31:37.785851 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.785830 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw"] Jul 28 12:31:37.786427 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.786411 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z"] Jul 28 12:31:37.901432 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.901408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e79aa138-009e-4652-a84b-b86654ca4b0b-tmp\") pod \"klusterlet-addon-workmgr-688fd47d7d-vj6cw\" (UID: \"e79aa138-009e-4652-a84b-b86654ca4b0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:31:37.901567 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.901476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsmtp\" (UniqueName: \"kubernetes.io/projected/e79aa138-009e-4652-a84b-b86654ca4b0b-kube-api-access-gsmtp\") pod \"klusterlet-addon-workmgr-688fd47d7d-vj6cw\" (UID: \"e79aa138-009e-4652-a84b-b86654ca4b0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:31:37.901567 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.901512 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655q2\" (UniqueName: \"kubernetes.io/projected/3f6f387b-b385-4048-bab2-0fe2d80e1ee9-kube-api-access-655q2\") pod \"managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z\" (UID: \"3f6f387b-b385-4048-bab2-0fe2d80e1ee9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z" Jul 28 12:31:37.901567 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.901545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e79aa138-009e-4652-a84b-b86654ca4b0b-klusterlet-config\") pod \"klusterlet-addon-workmgr-688fd47d7d-vj6cw\" (UID: \"e79aa138-009e-4652-a84b-b86654ca4b0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:31:37.901664 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:37.901568 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f6f387b-b385-4048-bab2-0fe2d80e1ee9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z\" (UID: \"3f6f387b-b385-4048-bab2-0fe2d80e1ee9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z" Jul 28 12:31:38.001813 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.001786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e79aa138-009e-4652-a84b-b86654ca4b0b-tmp\") pod \"klusterlet-addon-workmgr-688fd47d7d-vj6cw\" (UID: \"e79aa138-009e-4652-a84b-b86654ca4b0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:31:38.001889 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.001833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmtp\" (UniqueName: \"kubernetes.io/projected/e79aa138-009e-4652-a84b-b86654ca4b0b-kube-api-access-gsmtp\") pod \"klusterlet-addon-workmgr-688fd47d7d-vj6cw\" (UID: \"e79aa138-009e-4652-a84b-b86654ca4b0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:31:38.001889 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.001862 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655q2\" (UniqueName: \"kubernetes.io/projected/3f6f387b-b385-4048-bab2-0fe2d80e1ee9-kube-api-access-655q2\") pod \"managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z\" (UID: \"3f6f387b-b385-4048-bab2-0fe2d80e1ee9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z" Jul 28 12:31:38.001957 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.001903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e79aa138-009e-4652-a84b-b86654ca4b0b-klusterlet-config\") pod \"klusterlet-addon-workmgr-688fd47d7d-vj6cw\" (UID: \"e79aa138-009e-4652-a84b-b86654ca4b0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:31:38.001957 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.001927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f6f387b-b385-4048-bab2-0fe2d80e1ee9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z\" (UID: \"3f6f387b-b385-4048-bab2-0fe2d80e1ee9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z" Jul 28 12:31:38.002228 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.002206 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e79aa138-009e-4652-a84b-b86654ca4b0b-tmp\") pod \"klusterlet-addon-workmgr-688fd47d7d-vj6cw\" (UID: \"e79aa138-009e-4652-a84b-b86654ca4b0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:31:38.004475 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.004432 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f6f387b-b385-4048-bab2-0fe2d80e1ee9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z\" (UID: \"3f6f387b-b385-4048-bab2-0fe2d80e1ee9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z" Jul 28 12:31:38.004548 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.004525 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e79aa138-009e-4652-a84b-b86654ca4b0b-klusterlet-config\") pod \"klusterlet-addon-workmgr-688fd47d7d-vj6cw\" (UID: \"e79aa138-009e-4652-a84b-b86654ca4b0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:31:38.018566 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.018517 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmtp\" (UniqueName: \"kubernetes.io/projected/e79aa138-009e-4652-a84b-b86654ca4b0b-kube-api-access-gsmtp\") pod \"klusterlet-addon-workmgr-688fd47d7d-vj6cw\" (UID: \"e79aa138-009e-4652-a84b-b86654ca4b0b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:31:38.021214 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.021197 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-655q2\" (UniqueName: \"kubernetes.io/projected/3f6f387b-b385-4048-bab2-0fe2d80e1ee9-kube-api-access-655q2\") pod \"managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z\" (UID: \"3f6f387b-b385-4048-bab2-0fe2d80e1ee9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z" Jul 28 12:31:38.081344 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.081324 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:31:38.101430 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.101404 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z" Jul 28 12:31:38.213544 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.213511 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw"] Jul 28 12:31:38.217297 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:31:38.217268 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79aa138_009e_4652_a84b_b86654ca4b0b.slice/crio-240e124825061cb5fa6c5c47967622ae501981540ef6b29b7bbb0e0e3ef10314 WatchSource:0}: Error finding container 240e124825061cb5fa6c5c47967622ae501981540ef6b29b7bbb0e0e3ef10314: Status 404 returned error can't find the container with id 240e124825061cb5fa6c5c47967622ae501981540ef6b29b7bbb0e0e3ef10314 Jul 28 12:31:38.232419 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.232394 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z"] Jul 28 12:31:38.236305 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:31:38.236279 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f6f387b_b385_4048_bab2_0fe2d80e1ee9.slice/crio-26b6aecb650a69bffa479fccae4149a4da5eb0415350cc02a3fa613f3504b0f4 WatchSource:0}: Error finding container 26b6aecb650a69bffa479fccae4149a4da5eb0415350cc02a3fa613f3504b0f4: Status 404 returned error can't find the container with id 26b6aecb650a69bffa479fccae4149a4da5eb0415350cc02a3fa613f3504b0f4 Jul 28 12:31:38.693038 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.693002 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z" event={"ID":"3f6f387b-b385-4048-bab2-0fe2d80e1ee9","Type":"ContainerStarted","Data":"26b6aecb650a69bffa479fccae4149a4da5eb0415350cc02a3fa613f3504b0f4"} Jul 28 12:31:38.693968 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:38.693946 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" event={"ID":"e79aa138-009e-4652-a84b-b86654ca4b0b","Type":"ContainerStarted","Data":"240e124825061cb5fa6c5c47967622ae501981540ef6b29b7bbb0e0e3ef10314"} Jul 28 12:31:43.712565 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:43.712527 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z" event={"ID":"3f6f387b-b385-4048-bab2-0fe2d80e1ee9","Type":"ContainerStarted","Data":"710bee770a601b21d479f9234db6355af54f5a653781f0fba025e2a833e32dd3"} Jul 28 12:31:43.713960 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:43.713940 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" event={"ID":"e79aa138-009e-4652-a84b-b86654ca4b0b","Type":"ContainerStarted","Data":"bf3072693de69bfa7c9fc5658ef42d8c2cdcea2f2e9a979d5fe9e96fb7441741"} Jul 28 12:31:43.714138 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:43.714127 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:31:43.715744 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:43.715726 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:31:43.730058 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:43.730014 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z" podStartSLOduration=2.32228288 podStartE2EDuration="6.730000727s" totalImagesPullingTime="4.407717847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:31:43.729584323 +0000 UTC m=+88.988875075" watchObservedRunningTime="2026-07-28 12:31:43.730000727 +0000 UTC m=+88.989291467" Jul 28 12:31:43.747693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:43.747649 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" podStartSLOduration=2.30973394 podStartE2EDuration="6.747633304s" totalImagesPullingTime="4.437899364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:31:43.74726287 +0000 UTC m=+89.006553636" watchObservedRunningTime="2026-07-28 12:31:43.747633304 +0000 UTC m=+89.006924080" Jul 28 12:31:54.022744 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:54.022704 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:31:54.023095 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:31:54.022821 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 12:31:54.023095 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:31:54.022876 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls podName:bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4 nodeName:}" failed. No retries permitted until 2026-07-28 12:32:58.022862601 +0000 UTC m=+163.282153319 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls") pod "dns-default-mfjqx" (UID: "bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4") : secret "dns-default-metrics-tls" not found Jul 28 12:31:57.668593 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:31:57.668559 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-564mm" Jul 28 12:32:25.136110 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:32:25.136055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:32:25.136728 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:32:25.136225 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 12:32:25.136728 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:32:25.136321 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs podName:a4868cb1-759e-4f1a-a614-165690edf4a3 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:27.136296854 +0000 UTC m=+252.395587574 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs") pod "network-metrics-daemon-csfdh" (UID: "a4868cb1-759e-4f1a-a614-165690edf4a3") : secret "metrics-daemon-secret" not found Jul 28 12:32:53.157052 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:32:53.156999 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mfjqx" podUID="bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4" Jul 28 12:32:53.284537 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:32:53.284496 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:32:53.878209 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:32:53.878181 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mfjqx" Jul 28 12:32:58.054772 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:32:58.054736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:32:58.055235 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:32:58.054843 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 12:32:58.055235 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:32:58.054904 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls podName:bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:00.054890161 +0000 UTC m=+285.314180880 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls") pod "dns-default-mfjqx" (UID: "bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4") : secret "dns-default-metrics-tls" not found Jul 28 12:33:07.262318 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:33:07.262235 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:33:43.000135 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:33:43.000062 2573 generic.go:356] "Generic (PLEG): container finished" podID="3f6f387b-b385-4048-bab2-0fe2d80e1ee9" containerID="710bee770a601b21d479f9234db6355af54f5a653781f0fba025e2a833e32dd3" exitCode=255 Jul 28 12:33:43.000583 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:33:43.000140 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z" event={"ID":"3f6f387b-b385-4048-bab2-0fe2d80e1ee9","Type":"ContainerDied","Data":"710bee770a601b21d479f9234db6355af54f5a653781f0fba025e2a833e32dd3"} Jul 28 12:33:43.001666 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:33:43.001638 2573 generic.go:356] "Generic (PLEG): container finished" podID="e79aa138-009e-4652-a84b-b86654ca4b0b" containerID="bf3072693de69bfa7c9fc5658ef42d8c2cdcea2f2e9a979d5fe9e96fb7441741" exitCode=1 Jul 28 12:33:43.001793 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:33:43.001691 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" event={"ID":"e79aa138-009e-4652-a84b-b86654ca4b0b","Type":"ContainerDied","Data":"bf3072693de69bfa7c9fc5658ef42d8c2cdcea2f2e9a979d5fe9e96fb7441741"} Jul 28 12:33:43.002001 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:33:43.001986 2573 scope.go:117] "RemoveContainer" containerID="bf3072693de69bfa7c9fc5658ef42d8c2cdcea2f2e9a979d5fe9e96fb7441741" Jul 28 12:33:43.006275 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:33:43.006257 2573 scope.go:117] "RemoveContainer" containerID="710bee770a601b21d479f9234db6355af54f5a653781f0fba025e2a833e32dd3" Jul 28 12:33:43.714261 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:33:43.714227 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:33:44.010610 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:33:44.010518 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b878bfdc4-rnb8z" event={"ID":"3f6f387b-b385-4048-bab2-0fe2d80e1ee9","Type":"ContainerStarted","Data":"d673d3675008ca0cb92dc2d0ac690575eab626d1a6613fc33278e46f28d572a1"} Jul 28 12:33:44.012213 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:33:44.012183 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" event={"ID":"e79aa138-009e-4652-a84b-b86654ca4b0b","Type":"ContainerStarted","Data":"4bee3cd90ebded09e5c1689d8943f7c1ea230b9ad5794adcc74329ecda8baab6"} Jul 28 12:33:44.012399 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:33:44.012385 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:33:44.012974 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:33:44.012957 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688fd47d7d-vj6cw" Jul 28 12:34:12.476614 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.476576 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4"] Jul 28 12:34:12.479425 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.479403 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" Jul 28 12:34:12.480365 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.480344 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4"] Jul 28 12:34:12.482690 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.482669 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 28 12:34:12.482985 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.482973 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:12.483701 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.483685 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 28 12:34:12.484167 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.484155 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 28 12:34:12.484210 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.484164 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qfmth\"" Jul 28 12:34:12.486610 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.486249 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 28 12:34:12.486610 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.486483 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 28 12:34:12.488432 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.488414 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 12:34:12.488540 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.488466 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gx9bm\"" Jul 28 12:34:12.489454 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.489412 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 12:34:12.489598 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.489575 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 28 12:34:12.495515 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.495495 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4"] Jul 28 12:34:12.502124 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.502104 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4"] Jul 28 12:34:12.527751 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.527728 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/971959e5-2ea2-4dde-8a74-43e61cfd6066-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4\" (UID: \"971959e5-2ea2-4dde-8a74-43e61cfd6066\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" Jul 28 12:34:12.527838 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.527756 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9556f668-494f-4d33-8bc4-a3e615e50f52-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:12.527838 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.527792 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dszf9\" (UniqueName: \"kubernetes.io/projected/9556f668-494f-4d33-8bc4-a3e615e50f52-kube-api-access-dszf9\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:12.527838 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.527810 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:12.527953 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.527874 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/971959e5-2ea2-4dde-8a74-43e61cfd6066-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4\" (UID: \"971959e5-2ea2-4dde-8a74-43e61cfd6066\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" Jul 28 12:34:12.527953 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.527900 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv4fs\" (UniqueName: \"kubernetes.io/projected/971959e5-2ea2-4dde-8a74-43e61cfd6066-kube-api-access-dv4fs\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4\" (UID: \"971959e5-2ea2-4dde-8a74-43e61cfd6066\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" Jul 28 12:34:12.628139 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.628110 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/971959e5-2ea2-4dde-8a74-43e61cfd6066-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4\" (UID: \"971959e5-2ea2-4dde-8a74-43e61cfd6066\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" Jul 28 12:34:12.628139 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.628141 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9556f668-494f-4d33-8bc4-a3e615e50f52-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:12.628383 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.628180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dszf9\" (UniqueName: \"kubernetes.io/projected/9556f668-494f-4d33-8bc4-a3e615e50f52-kube-api-access-dszf9\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:12.628383 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.628203 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:12.628383 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.628241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/971959e5-2ea2-4dde-8a74-43e61cfd6066-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4\" (UID: \"971959e5-2ea2-4dde-8a74-43e61cfd6066\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" Jul 28 12:34:12.628383 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.628266 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv4fs\" (UniqueName: \"kubernetes.io/projected/971959e5-2ea2-4dde-8a74-43e61cfd6066-kube-api-access-dv4fs\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4\" (UID: \"971959e5-2ea2-4dde-8a74-43e61cfd6066\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" Jul 28 12:34:12.628383 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:12.628367 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 12:34:12.628664 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:12.628486 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls podName:9556f668-494f-4d33-8bc4-a3e615e50f52 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:13.12846278 +0000 UTC m=+238.387753501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-ldvw4" (UID: "9556f668-494f-4d33-8bc4-a3e615e50f52") : secret "cluster-monitoring-operator-tls" not found Jul 28 12:34:12.628727 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.628690 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/971959e5-2ea2-4dde-8a74-43e61cfd6066-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4\" (UID: \"971959e5-2ea2-4dde-8a74-43e61cfd6066\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" Jul 28 12:34:12.628935 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.628918 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9556f668-494f-4d33-8bc4-a3e615e50f52-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:12.630513 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.630495 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/971959e5-2ea2-4dde-8a74-43e61cfd6066-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4\" (UID: \"971959e5-2ea2-4dde-8a74-43e61cfd6066\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" Jul 28 12:34:12.636391 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.636369 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv4fs\" (UniqueName: \"kubernetes.io/projected/971959e5-2ea2-4dde-8a74-43e61cfd6066-kube-api-access-dv4fs\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4\" (UID: \"971959e5-2ea2-4dde-8a74-43e61cfd6066\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" Jul 28 12:34:12.636510 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.636423 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dszf9\" (UniqueName: \"kubernetes.io/projected/9556f668-494f-4d33-8bc4-a3e615e50f52-kube-api-access-dszf9\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:12.791195 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.791115 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" Jul 28 12:34:12.910296 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:12.910264 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4"] Jul 28 12:34:12.912802 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:34:12.912777 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971959e5_2ea2_4dde_8a74_43e61cfd6066.slice/crio-b94f68fdfe4c0b36690e80429c54f1853940114092d934635d688fe7bbf3a1de WatchSource:0}: Error finding container b94f68fdfe4c0b36690e80429c54f1853940114092d934635d688fe7bbf3a1de: Status 404 returned error can't find the container with id b94f68fdfe4c0b36690e80429c54f1853940114092d934635d688fe7bbf3a1de Jul 28 12:34:13.096421 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:13.096350 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" event={"ID":"971959e5-2ea2-4dde-8a74-43e61cfd6066","Type":"ContainerStarted","Data":"b94f68fdfe4c0b36690e80429c54f1853940114092d934635d688fe7bbf3a1de"} Jul 28 12:34:13.131840 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:13.131813 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:13.131936 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:13.131924 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 12:34:13.132005 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:13.131995 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls podName:9556f668-494f-4d33-8bc4-a3e615e50f52 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:14.131974956 +0000 UTC m=+239.391265695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-ldvw4" (UID: "9556f668-494f-4d33-8bc4-a3e615e50f52") : secret "cluster-monitoring-operator-tls" not found Jul 28 12:34:14.138707 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:14.138679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:14.139245 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:14.138815 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 12:34:14.139245 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:14.138870 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls podName:9556f668-494f-4d33-8bc4-a3e615e50f52 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:16.138855072 +0000 UTC m=+241.398145791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-ldvw4" (UID: "9556f668-494f-4d33-8bc4-a3e615e50f52") : secret "cluster-monitoring-operator-tls" not found Jul 28 12:34:15.102608 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:15.102526 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" event={"ID":"971959e5-2ea2-4dde-8a74-43e61cfd6066","Type":"ContainerStarted","Data":"3af1dae676b7ffed03e928d03313a01850d6044995402ee148b2380f2f35ac97"} Jul 28 12:34:15.119929 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:15.119887 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" podStartSLOduration=1.251477211 podStartE2EDuration="3.119871997s" totalImagesPullingTime="1.868394786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:15.118772537 +0000 UTC m=+240.378063278" watchObservedRunningTime="2026-07-28 12:34:15.119871997 +0000 UTC m=+240.379162738" Jul 28 12:34:16.153659 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.153625 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:16.154032 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:16.153814 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 12:34:16.154032 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:16.153888 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls podName:9556f668-494f-4d33-8bc4-a3e615e50f52 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:20.153868682 +0000 UTC m=+245.413159401 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-ldvw4" (UID: "9556f668-494f-4d33-8bc4-a3e615e50f52") : secret "cluster-monitoring-operator-tls" not found Jul 28 12:34:16.437337 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.437266 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-kc66k"] Jul 28 12:34:16.440330 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.440307 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-kc66k" Jul 28 12:34:16.443276 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.443247 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zjwqb\"" Jul 28 12:34:16.443376 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.443355 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 28 12:34:16.444486 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.444471 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 28 12:34:16.450045 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.450026 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-kc66k"] Jul 28 12:34:16.556328 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.556303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2krr\" (UniqueName: \"kubernetes.io/projected/1b859f33-8b1d-4d81-b58f-da7a845c9d57-kube-api-access-k2krr\") pod \"migrator-6865c9967-kc66k\" (UID: \"1b859f33-8b1d-4d81-b58f-da7a845c9d57\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-kc66k" Jul 28 12:34:16.644067 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.644040 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l7h6h"] Jul 28 12:34:16.647071 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.647056 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.649814 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.649787 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 12:34:16.650098 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.650083 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 12:34:16.650430 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.650415 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 12:34:16.650734 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.650715 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 12:34:16.651147 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.651133 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cv5n6\"" Jul 28 12:34:16.657201 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.657182 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2krr\" (UniqueName: \"kubernetes.io/projected/1b859f33-8b1d-4d81-b58f-da7a845c9d57-kube-api-access-k2krr\") pod \"migrator-6865c9967-kc66k\" (UID: \"1b859f33-8b1d-4d81-b58f-da7a845c9d57\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-kc66k" Jul 28 12:34:16.658609 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.658588 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l7h6h"] Jul 28 12:34:16.669376 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.669358 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2krr\" (UniqueName: \"kubernetes.io/projected/1b859f33-8b1d-4d81-b58f-da7a845c9d57-kube-api-access-k2krr\") pod \"migrator-6865c9967-kc66k\" (UID: \"1b859f33-8b1d-4d81-b58f-da7a845c9d57\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-kc66k" Jul 28 12:34:16.749643 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.749587 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-kc66k" Jul 28 12:34:16.757624 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.757598 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.757716 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.757653 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4ks5\" (UniqueName: \"kubernetes.io/projected/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-api-access-c4ks5\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.757773 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.757720 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dba89490-b79c-47bb-b22c-2d4d923e5e5c-crio-socket\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.757773 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.757743 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dba89490-b79c-47bb-b22c-2d4d923e5e5c-data-volume\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.757878 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.757803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.858983 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.858954 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ks5\" (UniqueName: \"kubernetes.io/projected/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-api-access-c4ks5\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.859122 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.859010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dba89490-b79c-47bb-b22c-2d4d923e5e5c-crio-socket\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.859122 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.859027 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dba89490-b79c-47bb-b22c-2d4d923e5e5c-data-volume\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.859122 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.859055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.859122 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.859102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.859308 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.859099 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dba89490-b79c-47bb-b22c-2d4d923e5e5c-crio-socket\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.859308 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:16.859178 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 12:34:16.859308 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:16.859232 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls podName:dba89490-b79c-47bb-b22c-2d4d923e5e5c nodeName:}" failed. No retries permitted until 2026-07-28 12:34:17.359218027 +0000 UTC m=+242.618508746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l7h6h" (UID: "dba89490-b79c-47bb-b22c-2d4d923e5e5c") : secret "insights-runtime-extractor-tls" not found Jul 28 12:34:16.859308 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:16.859176 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 12:34:16.859518 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:16.859336 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm podName:dba89490-b79c-47bb-b22c-2d4d923e5e5c nodeName:}" failed. No retries permitted until 2026-07-28 12:34:17.359317352 +0000 UTC m=+242.618608084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l7h6h" (UID: "dba89490-b79c-47bb-b22c-2d4d923e5e5c") : configmap "kube-rbac-proxy" not found Jul 28 12:34:16.859518 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.859418 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dba89490-b79c-47bb-b22c-2d4d923e5e5c-data-volume\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.869178 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.869153 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ks5\" (UniqueName: \"kubernetes.io/projected/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-api-access-c4ks5\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:16.869380 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:16.869364 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-kc66k"] Jul 28 12:34:16.872113 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:34:16.872080 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b859f33_8b1d_4d81_b58f_da7a845c9d57.slice/crio-46045d236807ca84382ed2c60a750d8a727d130831d6d0786f4c5947ea0d4263 WatchSource:0}: Error finding container 46045d236807ca84382ed2c60a750d8a727d130831d6d0786f4c5947ea0d4263: Status 404 returned error can't find the container with id 46045d236807ca84382ed2c60a750d8a727d130831d6d0786f4c5947ea0d4263 Jul 28 12:34:17.110189 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:17.110121 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-kc66k" event={"ID":"1b859f33-8b1d-4d81-b58f-da7a845c9d57","Type":"ContainerStarted","Data":"46045d236807ca84382ed2c60a750d8a727d130831d6d0786f4c5947ea0d4263"} Jul 28 12:34:17.363474 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:17.363394 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:17.363474 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:17.363464 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:17.363831 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:17.363547 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 12:34:17.363831 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:17.363548 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 12:34:17.363831 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:17.363603 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls podName:dba89490-b79c-47bb-b22c-2d4d923e5e5c nodeName:}" failed. No retries permitted until 2026-07-28 12:34:18.363588175 +0000 UTC m=+243.622878895 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l7h6h" (UID: "dba89490-b79c-47bb-b22c-2d4d923e5e5c") : secret "insights-runtime-extractor-tls" not found Jul 28 12:34:17.363831 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:17.363618 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm podName:dba89490-b79c-47bb-b22c-2d4d923e5e5c nodeName:}" failed. No retries permitted until 2026-07-28 12:34:18.363611825 +0000 UTC m=+243.622902544 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l7h6h" (UID: "dba89490-b79c-47bb-b22c-2d4d923e5e5c") : configmap "kube-rbac-proxy" not found Jul 28 12:34:18.372473 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:18.372431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:18.372827 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:18.372563 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 12:34:18.372827 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:18.372620 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:18.372827 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:18.372665 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 12:34:18.372827 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:18.372753 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls podName:dba89490-b79c-47bb-b22c-2d4d923e5e5c nodeName:}" failed. No retries permitted until 2026-07-28 12:34:20.37271057 +0000 UTC m=+245.632001289 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l7h6h" (UID: "dba89490-b79c-47bb-b22c-2d4d923e5e5c") : secret "insights-runtime-extractor-tls" not found Jul 28 12:34:18.372827 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:18.372776 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm podName:dba89490-b79c-47bb-b22c-2d4d923e5e5c nodeName:}" failed. No retries permitted until 2026-07-28 12:34:20.372763663 +0000 UTC m=+245.632054384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l7h6h" (UID: "dba89490-b79c-47bb-b22c-2d4d923e5e5c") : configmap "kube-rbac-proxy" not found Jul 28 12:34:19.117229 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:19.117179 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-kc66k" event={"ID":"1b859f33-8b1d-4d81-b58f-da7a845c9d57","Type":"ContainerStarted","Data":"5146e97e0bd2c013aabe5985222fe2eae4e92278920da9d98e162b0540c75f0b"} Jul 28 12:34:19.117396 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:19.117247 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-kc66k" event={"ID":"1b859f33-8b1d-4d81-b58f-da7a845c9d57","Type":"ContainerStarted","Data":"a8236424e404cef68d2de888ac7047d1bd9d6a44243529edfde1b951b0b7ad86"} Jul 28 12:34:19.134391 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:19.134343 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-kc66k" podStartSLOduration=1.86444018 podStartE2EDuration="3.134327701s" totalImagesPullingTime="1.269887521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:19.133857805 +0000 UTC m=+244.393148555" watchObservedRunningTime="2026-07-28 12:34:19.134327701 +0000 UTC m=+244.393618442" Jul 28 12:34:19.427335 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:19.427311 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wd7wp_8325bc39-4cb3-49bb-b803-18c2ab4a24e7/dns-node-resolver/0.log" Jul 28 12:34:20.184298 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:20.184261 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:20.184493 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:20.184386 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 12:34:20.184493 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:20.184474 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls podName:9556f668-494f-4d33-8bc4-a3e615e50f52 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:28.18443181 +0000 UTC m=+253.443722535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-ldvw4" (UID: "9556f668-494f-4d33-8bc4-a3e615e50f52") : secret "cluster-monitoring-operator-tls" not found Jul 28 12:34:20.386710 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:20.386674 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:20.386876 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:20.386756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:20.386876 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:20.386840 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 12:34:20.386876 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:20.386867 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 12:34:20.386976 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:20.386903 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls podName:dba89490-b79c-47bb-b22c-2d4d923e5e5c nodeName:}" failed. No retries permitted until 2026-07-28 12:34:24.386888672 +0000 UTC m=+249.646179391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l7h6h" (UID: "dba89490-b79c-47bb-b22c-2d4d923e5e5c") : secret "insights-runtime-extractor-tls" not found Jul 28 12:34:20.386976 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:20.386917 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm podName:dba89490-b79c-47bb-b22c-2d4d923e5e5c nodeName:}" failed. No retries permitted until 2026-07-28 12:34:24.386910635 +0000 UTC m=+249.646201354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l7h6h" (UID: "dba89490-b79c-47bb-b22c-2d4d923e5e5c") : configmap "kube-rbac-proxy" not found Jul 28 12:34:20.627544 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:20.627459 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vthpv_f2af6702-608a-497a-bcc5-d7eac4802553/node-ca/0.log" Jul 28 12:34:21.228894 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:21.228864 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qfsc7_4a764e09-ea01-4cff-a9b1-0b14c995b676/serve-healthcheck-canary/0.log" Jul 28 12:34:21.630647 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:21.630578 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-kc66k_1b859f33-8b1d-4d81-b58f-da7a845c9d57/migrator/0.log" Jul 28 12:34:21.829480 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:21.829456 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-kc66k_1b859f33-8b1d-4d81-b58f-da7a845c9d57/graceful-termination/0.log" Jul 28 12:34:22.028696 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:22.028668 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4_971959e5-2ea2-4dde-8a74-43e61cfd6066/kube-storage-version-migrator-operator/0.log" Jul 28 12:34:24.420213 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:24.420174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:24.420649 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:24.420244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:24.420649 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:24.420321 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 12:34:24.420649 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:24.420339 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 12:34:24.420649 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:24.420385 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls podName:dba89490-b79c-47bb-b22c-2d4d923e5e5c nodeName:}" failed. No retries permitted until 2026-07-28 12:34:32.420370198 +0000 UTC m=+257.679660917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l7h6h" (UID: "dba89490-b79c-47bb-b22c-2d4d923e5e5c") : secret "insights-runtime-extractor-tls" not found Jul 28 12:34:24.420649 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:24.420399 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm podName:dba89490-b79c-47bb-b22c-2d4d923e5e5c nodeName:}" failed. No retries permitted until 2026-07-28 12:34:32.420393792 +0000 UTC m=+257.679684510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l7h6h" (UID: "dba89490-b79c-47bb-b22c-2d4d923e5e5c") : configmap "kube-rbac-proxy" not found Jul 28 12:34:27.143279 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:27.143248 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:34:27.143692 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:27.143391 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 12:34:27.143692 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:27.143470 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs podName:a4868cb1-759e-4f1a-a614-165690edf4a3 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:29.143449846 +0000 UTC m=+374.402740578 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs") pod "network-metrics-daemon-csfdh" (UID: "a4868cb1-759e-4f1a-a614-165690edf4a3") : secret "metrics-daemon-secret" not found Jul 28 12:34:28.251411 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:28.251374 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:28.251897 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:28.251529 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 12:34:28.251897 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:28.251603 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls podName:9556f668-494f-4d33-8bc4-a3e615e50f52 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:44.251583879 +0000 UTC m=+269.510874600 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-ldvw4" (UID: "9556f668-494f-4d33-8bc4-a3e615e50f52") : secret "cluster-monitoring-operator-tls" not found Jul 28 12:34:32.482755 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:32.482709 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:32.483227 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:32.482780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:32.483227 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:32.482877 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 12:34:32.483227 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:32.482924 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls podName:dba89490-b79c-47bb-b22c-2d4d923e5e5c nodeName:}" failed. No retries permitted until 2026-07-28 12:34:48.482910795 +0000 UTC m=+273.742201514 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l7h6h" (UID: "dba89490-b79c-47bb-b22c-2d4d923e5e5c") : secret "insights-runtime-extractor-tls" not found Jul 28 12:34:32.483396 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:32.483303 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dba89490-b79c-47bb-b22c-2d4d923e5e5c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:44.262532 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:44.262432 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:44.264916 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:44.264896 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9556f668-494f-4d33-8bc4-a3e615e50f52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-ldvw4\" (UID: \"9556f668-494f-4d33-8bc4-a3e615e50f52\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:44.299867 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:44.299837 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gx9bm\"" Jul 28 12:34:44.307596 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:44.307574 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" Jul 28 12:34:44.434867 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:44.434845 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4"] Jul 28 12:34:44.437378 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:34:44.437351 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9556f668_494f_4d33_8bc4_a3e615e50f52.slice/crio-7c5648594bdd29c415a84c6eeb0e4e1592f2511e95924ab95544cee5618523c2 WatchSource:0}: Error finding container 7c5648594bdd29c415a84c6eeb0e4e1592f2511e95924ab95544cee5618523c2: Status 404 returned error can't find the container with id 7c5648594bdd29c415a84c6eeb0e4e1592f2511e95924ab95544cee5618523c2 Jul 28 12:34:45.146559 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.146526 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9d9774ff7-zfb87"] Jul 28 12:34:45.154162 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.154135 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.157223 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.157200 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c2wj5\"" Jul 28 12:34:45.157347 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.157278 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 12:34:45.158931 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.158787 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 12:34:45.158931 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.158826 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 12:34:45.158931 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.158849 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 12:34:45.158931 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.158909 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 12:34:45.159167 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.158946 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 12:34:45.159219 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.159205 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 12:34:45.161790 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.161764 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d9774ff7-zfb87"] Jul 28 12:34:45.195876 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.195853 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" event={"ID":"9556f668-494f-4d33-8bc4-a3e615e50f52","Type":"ContainerStarted","Data":"7c5648594bdd29c415a84c6eeb0e4e1592f2511e95924ab95544cee5618523c2"} Jul 28 12:34:45.268373 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.268348 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvb9\" (UniqueName: \"kubernetes.io/projected/d48896cf-9240-4225-8d7c-a68dc89612ce-kube-api-access-tpvb9\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.268736 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.268386 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-console-config\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.268736 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.268417 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-oauth-serving-cert\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.268736 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.268473 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-service-ca\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.268736 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.268550 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d48896cf-9240-4225-8d7c-a68dc89612ce-console-oauth-config\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.268736 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.268588 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d48896cf-9240-4225-8d7c-a68dc89612ce-console-serving-cert\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.369516 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.369489 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d48896cf-9240-4225-8d7c-a68dc89612ce-console-oauth-config\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.369660 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.369527 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d48896cf-9240-4225-8d7c-a68dc89612ce-console-serving-cert\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.369660 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.369608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvb9\" (UniqueName: \"kubernetes.io/projected/d48896cf-9240-4225-8d7c-a68dc89612ce-kube-api-access-tpvb9\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.369660 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.369632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-console-config\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.369811 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.369668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-oauth-serving-cert\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.369811 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.369699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-service-ca\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.370413 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.370391 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-console-config\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.370539 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.370391 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-service-ca\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.370651 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.370626 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-oauth-serving-cert\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.372208 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.372186 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d48896cf-9240-4225-8d7c-a68dc89612ce-console-oauth-config\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.372608 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.372588 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d48896cf-9240-4225-8d7c-a68dc89612ce-console-serving-cert\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.378571 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.378533 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvb9\" (UniqueName: \"kubernetes.io/projected/d48896cf-9240-4225-8d7c-a68dc89612ce-kube-api-access-tpvb9\") pod \"console-9d9774ff7-zfb87\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.464610 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.464547 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:45.600379 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:45.600223 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d9774ff7-zfb87"] Jul 28 12:34:45.603114 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:34:45.603082 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48896cf_9240_4225_8d7c_a68dc89612ce.slice/crio-c36caeecbd66d135f246396e6c28fdedbf4f73731d999b34e80bd5148fdbd0ca WatchSource:0}: Error finding container c36caeecbd66d135f246396e6c28fdedbf4f73731d999b34e80bd5148fdbd0ca: Status 404 returned error can't find the container with id c36caeecbd66d135f246396e6c28fdedbf4f73731d999b34e80bd5148fdbd0ca Jul 28 12:34:46.200512 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:46.200474 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d9774ff7-zfb87" event={"ID":"d48896cf-9240-4225-8d7c-a68dc89612ce","Type":"ContainerStarted","Data":"c36caeecbd66d135f246396e6c28fdedbf4f73731d999b34e80bd5148fdbd0ca"} Jul 28 12:34:46.962647 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:46.961807 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx"] Jul 28 12:34:46.965048 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:46.965014 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx" Jul 28 12:34:46.968940 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:46.968909 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 28 12:34:46.969059 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:46.968966 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-69twx\"" Jul 28 12:34:46.974562 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:46.974506 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx"] Jul 28 12:34:47.085837 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:47.085798 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/993c23fc-72c5-4a38-a973-41c653c2e958-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8njsx\" (UID: \"993c23fc-72c5-4a38-a973-41c653c2e958\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx" Jul 28 12:34:47.187242 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:47.187198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/993c23fc-72c5-4a38-a973-41c653c2e958-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8njsx\" (UID: \"993c23fc-72c5-4a38-a973-41c653c2e958\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx" Jul 28 12:34:47.187406 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:47.187326 2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 28 12:34:47.187406 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:47.187397 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/993c23fc-72c5-4a38-a973-41c653c2e958-tls-certificates podName:993c23fc-72c5-4a38-a973-41c653c2e958 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:47.687376602 +0000 UTC m=+272.946667326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/993c23fc-72c5-4a38-a973-41c653c2e958-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-8njsx" (UID: "993c23fc-72c5-4a38-a973-41c653c2e958") : secret "prometheus-operator-admission-webhook-tls" not found Jul 28 12:34:47.206433 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:47.206400 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" event={"ID":"9556f668-494f-4d33-8bc4-a3e615e50f52","Type":"ContainerStarted","Data":"8e30803a8f9ea0a086e54093b44cdc57df37b3ebcc413e797738faca47eefb8c"} Jul 28 12:34:47.691996 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:47.691875 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/993c23fc-72c5-4a38-a973-41c653c2e958-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8njsx\" (UID: \"993c23fc-72c5-4a38-a973-41c653c2e958\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx" Jul 28 12:34:47.697429 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:47.697400 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/993c23fc-72c5-4a38-a973-41c653c2e958-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8njsx\" (UID: \"993c23fc-72c5-4a38-a973-41c653c2e958\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx" Jul 28 12:34:47.881336 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:47.881303 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx" Jul 28 12:34:48.017685 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:48.017492 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-ldvw4" podStartSLOduration=34.016357562 podStartE2EDuration="36.017473208s" totalImagesPullingTime="2.001115646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:47.234373359 +0000 UTC m=+272.493664110" watchObservedRunningTime="2026-07-28 12:34:48.017473208 +0000 UTC m=+273.276763950" Jul 28 12:34:48.018060 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:48.017928 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx"] Jul 28 12:34:48.499735 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:48.499701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:48.502479 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:48.502453 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dba89490-b79c-47bb-b22c-2d4d923e5e5c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l7h6h\" (UID: \"dba89490-b79c-47bb-b22c-2d4d923e5e5c\") " pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:48.755386 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:48.755310 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l7h6h" Jul 28 12:34:48.978147 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:34:48.978110 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993c23fc_72c5_4a38_a973_41c653c2e958.slice/crio-3321feb9ed65330e658a91eb22a19fe7e261b02993b2b5cfa1ed7e3e231b5cf1 WatchSource:0}: Error finding container 3321feb9ed65330e658a91eb22a19fe7e261b02993b2b5cfa1ed7e3e231b5cf1: Status 404 returned error can't find the container with id 3321feb9ed65330e658a91eb22a19fe7e261b02993b2b5cfa1ed7e3e231b5cf1 Jul 28 12:34:49.101829 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:49.101805 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l7h6h"] Jul 28 12:34:49.103376 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:34:49.103350 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba89490_b79c_47bb_b22c_2d4d923e5e5c.slice/crio-35abb97ad314c1b98b596d0cb3d17d57887cbd5f3087db3dfafdf50ff0a88fc0 WatchSource:0}: Error finding container 35abb97ad314c1b98b596d0cb3d17d57887cbd5f3087db3dfafdf50ff0a88fc0: Status 404 returned error can't find the container with id 35abb97ad314c1b98b596d0cb3d17d57887cbd5f3087db3dfafdf50ff0a88fc0 Jul 28 12:34:49.215581 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:49.215542 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d9774ff7-zfb87" event={"ID":"d48896cf-9240-4225-8d7c-a68dc89612ce","Type":"ContainerStarted","Data":"75cc97222f1735c2101044de931d465374e2333dfd073204c9fcedd5ceba6f12"} Jul 28 12:34:49.216633 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:49.216612 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l7h6h" event={"ID":"dba89490-b79c-47bb-b22c-2d4d923e5e5c","Type":"ContainerStarted","Data":"35abb97ad314c1b98b596d0cb3d17d57887cbd5f3087db3dfafdf50ff0a88fc0"} Jul 28 12:34:49.217642 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:49.217621 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx" event={"ID":"993c23fc-72c5-4a38-a973-41c653c2e958","Type":"ContainerStarted","Data":"3321feb9ed65330e658a91eb22a19fe7e261b02993b2b5cfa1ed7e3e231b5cf1"} Jul 28 12:34:49.232943 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:49.232903 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9d9774ff7-zfb87" podStartSLOduration=0.801911591 podStartE2EDuration="4.232884258s" totalImagesPullingTime="3.430972667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:49.231822293 +0000 UTC m=+274.491113034" watchObservedRunningTime="2026-07-28 12:34:49.232884258 +0000 UTC m=+274.492174999" Jul 28 12:34:51.227596 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:51.227556 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx" event={"ID":"993c23fc-72c5-4a38-a973-41c653c2e958","Type":"ContainerStarted","Data":"147d76185f67c32190fb00b5dd8b086adb3a4bc2caceca2fae4333a0d8445fad"} Jul 28 12:34:51.228031 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:51.227888 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx" Jul 28 12:34:51.234028 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:51.234006 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx" Jul 28 12:34:51.247159 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:51.247108 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8njsx" podStartSLOduration=3.905645877 podStartE2EDuration="5.24708833s" totalImagesPullingTime="1.341442453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:51.244480956 +0000 UTC m=+276.503771716" watchObservedRunningTime="2026-07-28 12:34:51.24708833 +0000 UTC m=+276.506379071" Jul 28 12:34:53.534065 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.534034 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d65b8bb-w5w5w"] Jul 28 12:34:53.540420 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.540393 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.549232 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.549205 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d65b8bb-w5w5w"] Jul 28 12:34:53.553321 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.553290 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 12:34:53.639347 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.639319 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-trusted-ca-bundle\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.639347 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.639358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-console-config\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.639578 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.639389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcd08263-e787-44d6-b106-0603f93f54eb-console-serving-cert\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.639578 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.639411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-service-ca\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.639578 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.639499 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcd08263-e787-44d6-b106-0603f93f54eb-console-oauth-config\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.639578 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.639567 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnddl\" (UniqueName: \"kubernetes.io/projected/dcd08263-e787-44d6-b106-0603f93f54eb-kube-api-access-bnddl\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.639738 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.639606 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-oauth-serving-cert\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.740752 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.740719 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-oauth-serving-cert\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.740895 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.740793 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-trusted-ca-bundle\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.740895 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.740814 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-console-config\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.740895 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.740832 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcd08263-e787-44d6-b106-0603f93f54eb-console-serving-cert\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.740895 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.740849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-service-ca\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.740895 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.740868 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcd08263-e787-44d6-b106-0603f93f54eb-console-oauth-config\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.741113 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.740897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnddl\" (UniqueName: \"kubernetes.io/projected/dcd08263-e787-44d6-b106-0603f93f54eb-kube-api-access-bnddl\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.741560 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.741513 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-oauth-serving-cert\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.741560 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.741513 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-service-ca\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.741709 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.741670 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-console-config\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.742224 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.742202 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-trusted-ca-bundle\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.743603 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.743583 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcd08263-e787-44d6-b106-0603f93f54eb-console-oauth-config\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.743690 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.743649 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcd08263-e787-44d6-b106-0603f93f54eb-console-serving-cert\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.749954 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.749936 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnddl\" (UniqueName: \"kubernetes.io/projected/dcd08263-e787-44d6-b106-0603f93f54eb-kube-api-access-bnddl\") pod \"console-6d65b8bb-w5w5w\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:53.853752 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:53.853663 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:34:54.207828 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:54.207805 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d65b8bb-w5w5w"] Jul 28 12:34:54.210468 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:34:54.210429 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd08263_e787_44d6_b106_0603f93f54eb.slice/crio-2e93186cc504eea42f83a3726dfe315a85239e6608ba8f3adc7463dc89e09bba WatchSource:0}: Error finding container 2e93186cc504eea42f83a3726dfe315a85239e6608ba8f3adc7463dc89e09bba: Status 404 returned error can't find the container with id 2e93186cc504eea42f83a3726dfe315a85239e6608ba8f3adc7463dc89e09bba Jul 28 12:34:54.240271 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:54.240234 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d65b8bb-w5w5w" event={"ID":"dcd08263-e787-44d6-b106-0603f93f54eb","Type":"ContainerStarted","Data":"2e93186cc504eea42f83a3726dfe315a85239e6608ba8f3adc7463dc89e09bba"} Jul 28 12:34:54.241720 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:54.241700 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l7h6h" event={"ID":"dba89490-b79c-47bb-b22c-2d4d923e5e5c","Type":"ContainerStarted","Data":"9a1d4884cc60c1cb15b53fb3acb59a7a77318cc220e3d4b59e0bcc3021a236c3"} Jul 28 12:34:55.247084 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:55.247044 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d65b8bb-w5w5w" event={"ID":"dcd08263-e787-44d6-b106-0603f93f54eb","Type":"ContainerStarted","Data":"be39b5002461eb1b066f9ef0c4ea1d9600244a9a35937be43dc06e354416e624"} Jul 28 12:34:55.266649 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:55.265726 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d65b8bb-w5w5w" podStartSLOduration=2.26570791 podStartE2EDuration="2.26570791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:55.264467376 +0000 UTC m=+280.523758119" watchObservedRunningTime="2026-07-28 12:34:55.26570791 +0000 UTC m=+280.524998652" Jul 28 12:34:55.464853 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:55.464821 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:55.464999 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:55.464882 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:55.471364 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:55.471337 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:56.255206 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:56.255172 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:34:56.879281 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:34:56.879231 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mfjqx" podUID="bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4" Jul 28 12:34:57.254670 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:57.254636 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mfjqx" Jul 28 12:34:58.260875 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:34:58.260835 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l7h6h" event={"ID":"dba89490-b79c-47bb-b22c-2d4d923e5e5c","Type":"ContainerStarted","Data":"7a31ec2ab1901bc770cd4b86a8f738bf46b89bf099e853228e506b91569084c7"} Jul 28 12:35:00.094249 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:00.094213 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:35:00.096930 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:00.096905 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4-metrics-tls\") pod \"dns-default-mfjqx\" (UID: \"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4\") " pod="openshift-dns/dns-default-mfjqx" Jul 28 12:35:00.258970 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:00.258943 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bstgx\"" Jul 28 12:35:00.265821 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:00.265789 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mfjqx" Jul 28 12:35:01.221979 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:01.221955 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mfjqx"] Jul 28 12:35:01.225627 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:35:01.225601 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4637d3_b8e1_4a3c_9f5b_8445e881c4c4.slice/crio-490c875a38ba40f01c544db389cb3e9492b7ac94c6c115bbfa8d27967ff3d029 WatchSource:0}: Error finding container 490c875a38ba40f01c544db389cb3e9492b7ac94c6c115bbfa8d27967ff3d029: Status 404 returned error can't find the container with id 490c875a38ba40f01c544db389cb3e9492b7ac94c6c115bbfa8d27967ff3d029 Jul 28 12:35:01.273848 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:01.273816 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l7h6h" event={"ID":"dba89490-b79c-47bb-b22c-2d4d923e5e5c","Type":"ContainerStarted","Data":"28bb72872cc1b684a68feadf4e4cc8ae40972fede144338c56fda7c1349abac2"} Jul 28 12:35:01.274933 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:01.274911 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mfjqx" event={"ID":"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4","Type":"ContainerStarted","Data":"490c875a38ba40f01c544db389cb3e9492b7ac94c6c115bbfa8d27967ff3d029"} Jul 28 12:35:01.291330 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:01.291279 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l7h6h" podStartSLOduration=33.372343342 podStartE2EDuration="45.29126039s" totalImagesPullingTime="11.918917048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:01.290471164 +0000 UTC m=+286.549761905" watchObservedRunningTime="2026-07-28 12:35:01.29126039 +0000 UTC m=+286.550551130" Jul 28 12:35:03.284310 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:03.284277 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mfjqx" event={"ID":"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4","Type":"ContainerStarted","Data":"868974e7fcfa221f47bdcb38afca697a043e309dbd40f50187f3641575f4b9c5"} Jul 28 12:35:03.284742 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:03.284327 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mfjqx" event={"ID":"bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4","Type":"ContainerStarted","Data":"e40e9baca43466025c87ddca4a93ff29bf849c33f62701d650fd0a28b0080d80"} Jul 28 12:35:03.284742 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:03.284357 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mfjqx" Jul 28 12:35:03.305927 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:03.305856 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mfjqx" podStartSLOduration=251.462505897 podStartE2EDuration="4m13.305839664s" totalImagesPullingTime="1.843333767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:03.304898681 +0000 UTC m=+288.564189422" watchObservedRunningTime="2026-07-28 12:35:03.305839664 +0000 UTC m=+288.565130405" Jul 28 12:35:03.854503 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:03.854467 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:35:03.854695 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:03.854531 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:35:03.859198 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:03.859173 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:35:04.292160 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:04.292131 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:35:04.337925 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:04.337889 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9d9774ff7-zfb87"] Jul 28 12:35:08.461849 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.461816 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8rssd"] Jul 28 12:35:08.464924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.464901 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.468236 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.468213 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 12:35:08.468388 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.468266 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 12:35:08.468597 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.468557 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xd6th\"" Jul 28 12:35:08.469934 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.469917 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 12:35:08.470053 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.469970 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 12:35:08.556559 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.556537 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-textfile\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.556660 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.556569 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5302424c-8d67-4228-a781-d3b6aeab2288-sys\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.556660 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.556591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5302424c-8d67-4228-a781-d3b6aeab2288-metrics-client-ca\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.556660 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.556628 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-tls\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.556660 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.556649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-accelerators-collector-config\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.556826 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.556673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5302424c-8d67-4228-a781-d3b6aeab2288-root\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.556826 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.556688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w24tm\" (UniqueName: \"kubernetes.io/projected/5302424c-8d67-4228-a781-d3b6aeab2288-kube-api-access-w24tm\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.556826 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.556705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.556826 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.556720 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-wtmp\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.657851 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.657822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5302424c-8d67-4228-a781-d3b6aeab2288-root\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.657999 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.657857 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w24tm\" (UniqueName: \"kubernetes.io/projected/5302424c-8d67-4228-a781-d3b6aeab2288-kube-api-access-w24tm\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.657999 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.657876 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.657999 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.657892 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-wtmp\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.657999 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.657912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-textfile\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.657999 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.657935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5302424c-8d67-4228-a781-d3b6aeab2288-sys\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.657999 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.657939 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5302424c-8d67-4228-a781-d3b6aeab2288-root\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.657999 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.657965 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5302424c-8d67-4228-a781-d3b6aeab2288-metrics-client-ca\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.657999 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.657996 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5302424c-8d67-4228-a781-d3b6aeab2288-sys\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.658375 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.658019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-tls\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.658375 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.658048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-accelerators-collector-config\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.658375 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.658067 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-wtmp\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.658375 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.658362 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-textfile\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.658618 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.658601 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5302424c-8d67-4228-a781-d3b6aeab2288-metrics-client-ca\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.658719 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.658700 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-accelerators-collector-config\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.660331 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.660314 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-tls\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.660377 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.660356 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5302424c-8d67-4228-a781-d3b6aeab2288-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.669193 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.669170 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w24tm\" (UniqueName: \"kubernetes.io/projected/5302424c-8d67-4228-a781-d3b6aeab2288-kube-api-access-w24tm\") pod \"node-exporter-8rssd\" (UID: \"5302424c-8d67-4228-a781-d3b6aeab2288\") " pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.774335 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:08.774238 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8rssd" Jul 28 12:35:08.786938 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:35:08.786906 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5302424c_8d67_4228_a781_d3b6aeab2288.slice/crio-71cadad3c16391144bf9b52a6584a846ee0e9579f4814cccc4d6e2c292ec49dc WatchSource:0}: Error finding container 71cadad3c16391144bf9b52a6584a846ee0e9579f4814cccc4d6e2c292ec49dc: Status 404 returned error can't find the container with id 71cadad3c16391144bf9b52a6584a846ee0e9579f4814cccc4d6e2c292ec49dc Jul 28 12:35:09.305147 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:09.305111 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rssd" event={"ID":"5302424c-8d67-4228-a781-d3b6aeab2288","Type":"ContainerStarted","Data":"71cadad3c16391144bf9b52a6584a846ee0e9579f4814cccc4d6e2c292ec49dc"} Jul 28 12:35:10.263162 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:35:10.263129 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-csfdh" podUID="a4868cb1-759e-4f1a-a614-165690edf4a3" Jul 28 12:35:10.310431 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.310405 2573 generic.go:356] "Generic (PLEG): container finished" podID="5302424c-8d67-4228-a781-d3b6aeab2288" containerID="7d4dfeba0b89e85809c9b6f4cb8a931379fa6f2472759264eff7c3e0476f7422" exitCode=0 Jul 28 12:35:10.310554 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.310492 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rssd" event={"ID":"5302424c-8d67-4228-a781-d3b6aeab2288","Type":"ContainerDied","Data":"7d4dfeba0b89e85809c9b6f4cb8a931379fa6f2472759264eff7c3e0476f7422"} Jul 28 12:35:10.377302 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.377267 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb"] Jul 28 12:35:10.380958 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.380935 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.383978 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.383951 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 28 12:35:10.383978 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.383971 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 28 12:35:10.384135 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.383982 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1ajhf8g9pn45b\"" Jul 28 12:35:10.384135 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.384031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 28 12:35:10.384135 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.384041 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 28 12:35:10.384370 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.384343 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 28 12:35:10.384424 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.384408 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9lwc7\"" Jul 28 12:35:10.390601 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.390573 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb"] Jul 28 12:35:10.471967 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.471939 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9szc\" (UniqueName: \"kubernetes.io/projected/50ceed3d-85bc-4237-a07e-6141140d1f72-kube-api-access-s9szc\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.472093 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.471974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.472093 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.472020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.472093 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.472038 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-grpc-tls\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.472093 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.472060 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.472093 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.472083 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.472259 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.472139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-tls\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.472259 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.472174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50ceed3d-85bc-4237-a07e-6141140d1f72-metrics-client-ca\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.572981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.572920 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.572981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.572947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.572981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.572970 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-tls\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.573151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.572989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50ceed3d-85bc-4237-a07e-6141140d1f72-metrics-client-ca\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.573151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.573050 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9szc\" (UniqueName: \"kubernetes.io/projected/50ceed3d-85bc-4237-a07e-6141140d1f72-kube-api-access-s9szc\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.573151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.573080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.573151 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.573131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.573343 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.573151 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-grpc-tls\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.574066 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.574041 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50ceed3d-85bc-4237-a07e-6141140d1f72-metrics-client-ca\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.575753 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.575728 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-tls\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.575868 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.575780 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.576188 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.576160 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.576278 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.576260 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-grpc-tls\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.576330 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.576293 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.576383 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.576353 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50ceed3d-85bc-4237-a07e-6141140d1f72-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.580860 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.580841 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9szc\" (UniqueName: \"kubernetes.io/projected/50ceed3d-85bc-4237-a07e-6141140d1f72-kube-api-access-s9szc\") pod \"thanos-querier-6ffb65cc44-wt7jb\" (UID: \"50ceed3d-85bc-4237-a07e-6141140d1f72\") " pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.717310 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.717276 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:10.851861 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:10.851797 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb"] Jul 28 12:35:10.855089 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:35:10.855054 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ceed3d_85bc_4237_a07e_6141140d1f72.slice/crio-9fb6b4d0f894ea6b7ebee47882ca6d0c1559036a244763fe2073e791f5471b78 WatchSource:0}: Error finding container 9fb6b4d0f894ea6b7ebee47882ca6d0c1559036a244763fe2073e791f5471b78: Status 404 returned error can't find the container with id 9fb6b4d0f894ea6b7ebee47882ca6d0c1559036a244763fe2073e791f5471b78 Jul 28 12:35:11.315607 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:11.315569 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rssd" event={"ID":"5302424c-8d67-4228-a781-d3b6aeab2288","Type":"ContainerStarted","Data":"4a8fa2e65f2d518dffab069f247a15280d6f6d662896fd35ac2ee4bb77a1d745"} Jul 28 12:35:11.316009 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:11.315622 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rssd" event={"ID":"5302424c-8d67-4228-a781-d3b6aeab2288","Type":"ContainerStarted","Data":"000e06952eb82d12591bfe6df82812513bcede10b696ff73642ae698c2ef5089"} Jul 28 12:35:11.316752 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:11.316729 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" event={"ID":"50ceed3d-85bc-4237-a07e-6141140d1f72","Type":"ContainerStarted","Data":"9fb6b4d0f894ea6b7ebee47882ca6d0c1559036a244763fe2073e791f5471b78"} Jul 28 12:35:11.337877 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:11.337833 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8rssd" podStartSLOduration=2.419624191 podStartE2EDuration="3.337818333s" totalImagesPullingTime="844.807438ms" totalInitContainerRuntime="73.386704ms" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:11.336591955 +0000 UTC m=+296.595882697" watchObservedRunningTime="2026-07-28 12:35:11.337818333 +0000 UTC m=+296.597109095" Jul 28 12:35:13.183120 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.183093 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74bd755f5-q7fbx"] Jul 28 12:35:13.186568 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.186538 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.196917 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.196897 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-service-ca\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.197027 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.196943 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-trusted-ca-bundle\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.197027 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.197004 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-oauth-serving-cert\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.197121 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.197030 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7f981e6-8e16-4920-9b3c-0808600fa625-console-oauth-config\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.197121 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.197057 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7f981e6-8e16-4920-9b3c-0808600fa625-console-serving-cert\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.197121 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.197082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-console-config\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.197121 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.197106 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr2hh\" (UniqueName: \"kubernetes.io/projected/a7f981e6-8e16-4920-9b3c-0808600fa625-kube-api-access-cr2hh\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.200178 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.200156 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74bd755f5-q7fbx"] Jul 28 12:35:13.290028 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.290009 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mfjqx" Jul 28 12:35:13.297933 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.297905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-trusted-ca-bundle\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.298150 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.297988 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-oauth-serving-cert\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.298150 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.298026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7f981e6-8e16-4920-9b3c-0808600fa625-console-oauth-config\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.298150 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.298050 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7f981e6-8e16-4920-9b3c-0808600fa625-console-serving-cert\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.298150 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.298079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-console-config\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.298150 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.298100 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr2hh\" (UniqueName: \"kubernetes.io/projected/a7f981e6-8e16-4920-9b3c-0808600fa625-kube-api-access-cr2hh\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.298150 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.298136 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-service-ca\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.298899 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.298853 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-trusted-ca-bundle\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.299215 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.299055 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-oauth-serving-cert\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.299281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.299239 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-service-ca\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.299533 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.299514 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-console-config\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.300808 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.300790 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7f981e6-8e16-4920-9b3c-0808600fa625-console-oauth-config\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.300981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.300958 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7f981e6-8e16-4920-9b3c-0808600fa625-console-serving-cert\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.306058 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.306037 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr2hh\" (UniqueName: \"kubernetes.io/projected/a7f981e6-8e16-4920-9b3c-0808600fa625-kube-api-access-cr2hh\") pod \"console-74bd755f5-q7fbx\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.327631 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.327604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" event={"ID":"50ceed3d-85bc-4237-a07e-6141140d1f72","Type":"ContainerStarted","Data":"48eba6454c6a0c5bf5ba36ab9d311f77677e9557f139ae4c2920b585ac80b2db"} Jul 28 12:35:13.327750 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.327679 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" event={"ID":"50ceed3d-85bc-4237-a07e-6141140d1f72","Type":"ContainerStarted","Data":"c900377de3704f3a86141d35ee0c6af1b16ae85a1feda87f19376cb64d227e28"} Jul 28 12:35:13.504908 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.504821 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:13.627950 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.627913 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq"] Jul 28 12:35:13.632239 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.632217 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.635368 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.635346 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 28 12:35:13.635501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.635347 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 28 12:35:13.635501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.635347 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 28 12:35:13.635501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.635415 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 28 12:35:13.635669 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.635507 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-459b2\"" Jul 28 12:35:13.635669 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.635527 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 28 12:35:13.641159 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.641142 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 28 12:35:13.648734 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.648717 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq"] Jul 28 12:35:13.649844 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.649821 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74bd755f5-q7fbx"] Jul 28 12:35:13.650091 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:35:13.650075 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f981e6_8e16_4920_9b3c_0808600fa625.slice/crio-25ebd461f0f66b223ee9ccf37cd930ba5aaf1cc21f44827dc5b9f7c22108370c WatchSource:0}: Error finding container 25ebd461f0f66b223ee9ccf37cd930ba5aaf1cc21f44827dc5b9f7c22108370c: Status 404 returned error can't find the container with id 25ebd461f0f66b223ee9ccf37cd930ba5aaf1cc21f44827dc5b9f7c22108370c Jul 28 12:35:13.702267 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.702243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-federate-client-tls\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.702359 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.702284 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.702401 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.702365 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-serving-certs-ca-bundle\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.702476 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.702402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-metrics-client-ca\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.702476 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.702431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrk6j\" (UniqueName: \"kubernetes.io/projected/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-kube-api-access-hrk6j\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.702558 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.702508 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.702661 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.702611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-telemeter-client-tls\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.702718 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.702672 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-secret-telemeter-client\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.803454 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.803355 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-telemeter-client-tls\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.803454 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.803404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-secret-telemeter-client\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.803454 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.803430 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-federate-client-tls\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.803723 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.803462 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.803723 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.803617 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-serving-certs-ca-bundle\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.803723 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.803650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-metrics-client-ca\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.803723 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.803675 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrk6j\" (UniqueName: \"kubernetes.io/projected/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-kube-api-access-hrk6j\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.803723 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.803717 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.804508 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.804481 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-metrics-client-ca\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.804624 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.804520 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-serving-certs-ca-bundle\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.804624 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.804539 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.806843 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.806817 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.806843 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.806832 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-telemeter-client-tls\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.806951 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.806882 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-federate-client-tls\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.806951 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.806906 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-secret-telemeter-client\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.812150 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.812132 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrk6j\" (UniqueName: \"kubernetes.io/projected/cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb-kube-api-access-hrk6j\") pod \"telemeter-client-5dbcbd6cf6-djpjq\" (UID: \"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb\") " pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:13.941820 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:13.941786 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" Jul 28 12:35:14.180700 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:14.180679 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq"] Jul 28 12:35:14.183475 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:35:14.183450 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1093eb_6e7c_4fae_9b9c_e0ed034c7acb.slice/crio-f85fc2a96f11acb59f83cf090e52a5dc9c85c4217f02deafb2622e3f5f8c7cae WatchSource:0}: Error finding container f85fc2a96f11acb59f83cf090e52a5dc9c85c4217f02deafb2622e3f5f8c7cae: Status 404 returned error can't find the container with id f85fc2a96f11acb59f83cf090e52a5dc9c85c4217f02deafb2622e3f5f8c7cae Jul 28 12:35:14.334744 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:14.334646 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" event={"ID":"50ceed3d-85bc-4237-a07e-6141140d1f72","Type":"ContainerStarted","Data":"148a06472cc37edbabeb7da0364c9798e09b657a136de5580bdc50876dd15daa"} Jul 28 12:35:14.334744 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:14.334717 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" event={"ID":"50ceed3d-85bc-4237-a07e-6141140d1f72","Type":"ContainerStarted","Data":"6b1e375f2f294b07d9ec04dbe2024a26afeede2c172f2233fc1cb72b7e85ea06"} Jul 28 12:35:14.334958 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:14.334757 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" event={"ID":"50ceed3d-85bc-4237-a07e-6141140d1f72","Type":"ContainerStarted","Data":"488c6e78811fcd5deccf5cc658a52474d77a286b164976f69232d63ec6655c56"} Jul 28 12:35:14.334958 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:14.334796 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" event={"ID":"50ceed3d-85bc-4237-a07e-6141140d1f72","Type":"ContainerStarted","Data":"4a9fc8ec297631b95763a614ed969af1809d9159ee92905b31ae9271490d949c"} Jul 28 12:35:14.334958 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:14.334831 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:14.335984 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:14.335951 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" event={"ID":"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb","Type":"ContainerStarted","Data":"f85fc2a96f11acb59f83cf090e52a5dc9c85c4217f02deafb2622e3f5f8c7cae"} Jul 28 12:35:14.337452 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:14.337420 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bd755f5-q7fbx" event={"ID":"a7f981e6-8e16-4920-9b3c-0808600fa625","Type":"ContainerStarted","Data":"6ccadc6956fb3fc05179296bc8080b9dbd5fc30b02169d241a05bd6f06d3fecf"} Jul 28 12:35:14.337538 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:14.337478 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bd755f5-q7fbx" event={"ID":"a7f981e6-8e16-4920-9b3c-0808600fa625","Type":"ContainerStarted","Data":"25ebd461f0f66b223ee9ccf37cd930ba5aaf1cc21f44827dc5b9f7c22108370c"} Jul 28 12:35:14.358051 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:14.358005 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" podStartSLOduration=1.323128567 podStartE2EDuration="4.357991785s" totalImagesPullingTime="3.034863218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:14.355786095 +0000 UTC m=+299.615076837" watchObservedRunningTime="2026-07-28 12:35:14.357991785 +0000 UTC m=+299.617282527" Jul 28 12:35:14.375252 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:14.375198 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74bd755f5-q7fbx" podStartSLOduration=1.375187077 podStartE2EDuration="1.375187077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:14.374097562 +0000 UTC m=+299.633388303" watchObservedRunningTime="2026-07-28 12:35:14.375187077 +0000 UTC m=+299.634477818" Jul 28 12:35:15.137390 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:15.137366 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 12:35:16.348050 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:16.347971 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" event={"ID":"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb","Type":"ContainerStarted","Data":"a56828fb8e18e66a4dc65ca65afd756cc9b5946f01ef3cd7b2f90c7ace9d6f61"} Jul 28 12:35:17.353779 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:17.353677 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" event={"ID":"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb","Type":"ContainerStarted","Data":"edb453c3be4dc28b9893af232d4eab2b19067a76775452798ecbb1977b57b62b"} Jul 28 12:35:17.353779 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:17.353739 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" event={"ID":"cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb","Type":"ContainerStarted","Data":"315115cb6e34802f2184422ccdcf26381d6c475bb539346364d5610515fd372b"} Jul 28 12:35:17.379139 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:17.379091 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5dbcbd6cf6-djpjq" podStartSLOduration=1.662094452 podStartE2EDuration="4.379078395s" totalImagesPullingTime="2.716983943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:17.378773822 +0000 UTC m=+302.638064565" watchObservedRunningTime="2026-07-28 12:35:17.379078395 +0000 UTC m=+302.638369136" Jul 28 12:35:18.895266 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:18.895228 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74bd755f5-q7fbx"] Jul 28 12:35:18.923731 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:18.923707 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f7787dbd8-7h5dx"] Jul 28 12:35:18.926909 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:18.926892 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:18.936554 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:18.936533 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f7787dbd8-7h5dx"] Jul 28 12:35:19.049678 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.049651 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-oauth-serving-cert\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.049834 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.049691 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/000c6e16-dd51-4e8e-8689-351387700681-console-serving-cert\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.049834 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.049717 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-service-ca\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.049834 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.049750 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-console-config\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.049834 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.049809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g58p\" (UniqueName: \"kubernetes.io/projected/000c6e16-dd51-4e8e-8689-351387700681-kube-api-access-2g58p\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.050010 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.049859 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-trusted-ca-bundle\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.050010 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.049906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/000c6e16-dd51-4e8e-8689-351387700681-console-oauth-config\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.151228 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.151141 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-service-ca\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.151228 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.151187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-console-config\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.151228 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.151216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g58p\" (UniqueName: \"kubernetes.io/projected/000c6e16-dd51-4e8e-8689-351387700681-kube-api-access-2g58p\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.151492 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.151389 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-trusted-ca-bundle\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.151557 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.151511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/000c6e16-dd51-4e8e-8689-351387700681-console-oauth-config\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.151603 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.151576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-oauth-serving-cert\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.151659 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.151624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/000c6e16-dd51-4e8e-8689-351387700681-console-serving-cert\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.152009 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.151971 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-service-ca\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.152115 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.152010 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-console-config\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.152172 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.152155 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-trusted-ca-bundle\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.152212 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.152180 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-oauth-serving-cert\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.153992 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.153965 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/000c6e16-dd51-4e8e-8689-351387700681-console-oauth-config\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.154191 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.154170 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/000c6e16-dd51-4e8e-8689-351387700681-console-serving-cert\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.159581 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.159563 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g58p\" (UniqueName: \"kubernetes.io/projected/000c6e16-dd51-4e8e-8689-351387700681-kube-api-access-2g58p\") pod \"console-f7787dbd8-7h5dx\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.235491 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.235467 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:19.363910 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:19.363887 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f7787dbd8-7h5dx"] Jul 28 12:35:19.365784 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:35:19.365759 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000c6e16_dd51_4e8e_8689_351387700681.slice/crio-b021aebbd2165948a32144522fe9703e023758d87bd645828f169309b0ddf418 WatchSource:0}: Error finding container b021aebbd2165948a32144522fe9703e023758d87bd645828f169309b0ddf418: Status 404 returned error can't find the container with id b021aebbd2165948a32144522fe9703e023758d87bd645828f169309b0ddf418 Jul 28 12:35:20.348382 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:20.348353 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6ffb65cc44-wt7jb" Jul 28 12:35:20.366898 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:20.366874 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7787dbd8-7h5dx" event={"ID":"000c6e16-dd51-4e8e-8689-351387700681","Type":"ContainerStarted","Data":"1474ea9b171a68dc5f4556de8fdeda29f0375b98617959af16ff6d3ebc9d4882"} Jul 28 12:35:20.367026 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:20.366911 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7787dbd8-7h5dx" event={"ID":"000c6e16-dd51-4e8e-8689-351387700681","Type":"ContainerStarted","Data":"b021aebbd2165948a32144522fe9703e023758d87bd645828f169309b0ddf418"} Jul 28 12:35:20.389407 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:20.389366 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f7787dbd8-7h5dx" podStartSLOduration=2.389352544 podStartE2EDuration="2.389352544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:20.387859839 +0000 UTC m=+305.647150580" watchObservedRunningTime="2026-07-28 12:35:20.389352544 +0000 UTC m=+305.648643337" Jul 28 12:35:23.262535 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:23.262495 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:35:23.505078 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:23.505043 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:29.236034 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.235992 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:29.236414 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.236062 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:29.240621 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.240595 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:29.358579 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.358520 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9d9774ff7-zfb87" podUID="d48896cf-9240-4225-8d7c-a68dc89612ce" containerName="console" containerID="cri-o://75cc97222f1735c2101044de931d465374e2333dfd073204c9fcedd5ceba6f12" gracePeriod=15 Jul 28 12:35:29.405015 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.404990 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:35:29.452224 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.452194 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d65b8bb-w5w5w"] Jul 28 12:35:29.618695 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.618669 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:35:29.746482 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.746396 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpvb9\" (UniqueName: \"kubernetes.io/projected/d48896cf-9240-4225-8d7c-a68dc89612ce-kube-api-access-tpvb9\") pod \"d48896cf-9240-4225-8d7c-a68dc89612ce\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " Jul 28 12:35:29.746681 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.746509 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d48896cf-9240-4225-8d7c-a68dc89612ce-console-oauth-config\") pod \"d48896cf-9240-4225-8d7c-a68dc89612ce\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " Jul 28 12:35:29.746681 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.746536 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-oauth-serving-cert\") pod \"d48896cf-9240-4225-8d7c-a68dc89612ce\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " Jul 28 12:35:29.746681 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.746557 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-service-ca\") pod \"d48896cf-9240-4225-8d7c-a68dc89612ce\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " Jul 28 12:35:29.746681 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.746586 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-console-config\") pod \"d48896cf-9240-4225-8d7c-a68dc89612ce\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " Jul 28 12:35:29.746681 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.746617 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d48896cf-9240-4225-8d7c-a68dc89612ce-console-serving-cert\") pod \"d48896cf-9240-4225-8d7c-a68dc89612ce\" (UID: \"d48896cf-9240-4225-8d7c-a68dc89612ce\") " Jul 28 12:35:29.746984 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.746961 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d48896cf-9240-4225-8d7c-a68dc89612ce" (UID: "d48896cf-9240-4225-8d7c-a68dc89612ce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:35:29.746984 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.746969 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-console-config" (OuterVolumeSpecName: "console-config") pod "d48896cf-9240-4225-8d7c-a68dc89612ce" (UID: "d48896cf-9240-4225-8d7c-a68dc89612ce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:35:29.747062 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.746979 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-service-ca" (OuterVolumeSpecName: "service-ca") pod "d48896cf-9240-4225-8d7c-a68dc89612ce" (UID: "d48896cf-9240-4225-8d7c-a68dc89612ce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:35:29.748919 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.748890 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48896cf-9240-4225-8d7c-a68dc89612ce-kube-api-access-tpvb9" (OuterVolumeSpecName: "kube-api-access-tpvb9") pod "d48896cf-9240-4225-8d7c-a68dc89612ce" (UID: "d48896cf-9240-4225-8d7c-a68dc89612ce"). InnerVolumeSpecName "kube-api-access-tpvb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:35:29.748919 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.748898 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48896cf-9240-4225-8d7c-a68dc89612ce-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d48896cf-9240-4225-8d7c-a68dc89612ce" (UID: "d48896cf-9240-4225-8d7c-a68dc89612ce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:35:29.748919 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.748916 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48896cf-9240-4225-8d7c-a68dc89612ce-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d48896cf-9240-4225-8d7c-a68dc89612ce" (UID: "d48896cf-9240-4225-8d7c-a68dc89612ce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:35:29.847851 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.847759 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpvb9\" (UniqueName: \"kubernetes.io/projected/d48896cf-9240-4225-8d7c-a68dc89612ce-kube-api-access-tpvb9\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:29.847851 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.847794 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d48896cf-9240-4225-8d7c-a68dc89612ce-console-oauth-config\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:29.847851 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.847804 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-oauth-serving-cert\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:29.847851 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.847812 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:29.847851 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.847821 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d48896cf-9240-4225-8d7c-a68dc89612ce-console-config\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:29.847851 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:29.847829 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d48896cf-9240-4225-8d7c-a68dc89612ce-console-serving-cert\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:30.406378 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:30.406341 2573 generic.go:356] "Generic (PLEG): container finished" podID="d48896cf-9240-4225-8d7c-a68dc89612ce" containerID="75cc97222f1735c2101044de931d465374e2333dfd073204c9fcedd5ceba6f12" exitCode=0 Jul 28 12:35:30.406888 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:30.406388 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d9774ff7-zfb87" Jul 28 12:35:30.406888 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:30.406426 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d9774ff7-zfb87" event={"ID":"d48896cf-9240-4225-8d7c-a68dc89612ce","Type":"ContainerDied","Data":"75cc97222f1735c2101044de931d465374e2333dfd073204c9fcedd5ceba6f12"} Jul 28 12:35:30.406888 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:30.406513 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d9774ff7-zfb87" event={"ID":"d48896cf-9240-4225-8d7c-a68dc89612ce","Type":"ContainerDied","Data":"c36caeecbd66d135f246396e6c28fdedbf4f73731d999b34e80bd5148fdbd0ca"} Jul 28 12:35:30.406888 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:30.406541 2573 scope.go:117] "RemoveContainer" containerID="75cc97222f1735c2101044de931d465374e2333dfd073204c9fcedd5ceba6f12" Jul 28 12:35:30.415309 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:30.415289 2573 scope.go:117] "RemoveContainer" containerID="75cc97222f1735c2101044de931d465374e2333dfd073204c9fcedd5ceba6f12" Jul 28 12:35:30.415604 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:35:30.415582 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75cc97222f1735c2101044de931d465374e2333dfd073204c9fcedd5ceba6f12\": container with ID starting with 75cc97222f1735c2101044de931d465374e2333dfd073204c9fcedd5ceba6f12 not found: ID does not exist" containerID="75cc97222f1735c2101044de931d465374e2333dfd073204c9fcedd5ceba6f12" Jul 28 12:35:30.415674 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:30.415612 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75cc97222f1735c2101044de931d465374e2333dfd073204c9fcedd5ceba6f12"} err="failed to get container status \"75cc97222f1735c2101044de931d465374e2333dfd073204c9fcedd5ceba6f12\": rpc error: code = NotFound desc = could not find container \"75cc97222f1735c2101044de931d465374e2333dfd073204c9fcedd5ceba6f12\": container with ID starting with 75cc97222f1735c2101044de931d465374e2333dfd073204c9fcedd5ceba6f12 not found: ID does not exist" Jul 28 12:35:30.430044 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:30.430015 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9d9774ff7-zfb87"] Jul 28 12:35:30.436122 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:30.436098 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9d9774ff7-zfb87"] Jul 28 12:35:31.267525 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:31.267494 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d48896cf-9240-4225-8d7c-a68dc89612ce" path="/var/lib/kubelet/pods/d48896cf-9240-4225-8d7c-a68dc89612ce/volumes" Jul 28 12:35:36.431467 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:36.431418 2573 generic.go:356] "Generic (PLEG): container finished" podID="971959e5-2ea2-4dde-8a74-43e61cfd6066" containerID="3af1dae676b7ffed03e928d03313a01850d6044995402ee148b2380f2f35ac97" exitCode=0 Jul 28 12:35:36.431884 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:36.431491 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" event={"ID":"971959e5-2ea2-4dde-8a74-43e61cfd6066","Type":"ContainerDied","Data":"3af1dae676b7ffed03e928d03313a01850d6044995402ee148b2380f2f35ac97"} Jul 28 12:35:36.431884 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:36.431788 2573 scope.go:117] "RemoveContainer" containerID="3af1dae676b7ffed03e928d03313a01850d6044995402ee148b2380f2f35ac97" Jul 28 12:35:37.440750 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:37.440720 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4" event={"ID":"971959e5-2ea2-4dde-8a74-43e61cfd6066","Type":"ContainerStarted","Data":"22cbf1224a60ff0626e9d8923be3d433d8cb02ce45e126373264f1a638a8de82"} Jul 28 12:35:43.914058 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:43.913996 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74bd755f5-q7fbx" podUID="a7f981e6-8e16-4920-9b3c-0808600fa625" containerName="console" containerID="cri-o://6ccadc6956fb3fc05179296bc8080b9dbd5fc30b02169d241a05bd6f06d3fecf" gracePeriod=15 Jul 28 12:35:44.156396 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.156369 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:44.260498 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.260416 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-console-config\") pod \"a7f981e6-8e16-4920-9b3c-0808600fa625\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " Jul 28 12:35:44.260648 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.260497 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7f981e6-8e16-4920-9b3c-0808600fa625-console-oauth-config\") pod \"a7f981e6-8e16-4920-9b3c-0808600fa625\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " Jul 28 12:35:44.260648 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.260528 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7f981e6-8e16-4920-9b3c-0808600fa625-console-serving-cert\") pod \"a7f981e6-8e16-4920-9b3c-0808600fa625\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " Jul 28 12:35:44.260648 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.260551 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-service-ca\") pod \"a7f981e6-8e16-4920-9b3c-0808600fa625\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " Jul 28 12:35:44.260648 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.260569 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-oauth-serving-cert\") pod \"a7f981e6-8e16-4920-9b3c-0808600fa625\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " Jul 28 12:35:44.260648 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.260606 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr2hh\" (UniqueName: \"kubernetes.io/projected/a7f981e6-8e16-4920-9b3c-0808600fa625-kube-api-access-cr2hh\") pod \"a7f981e6-8e16-4920-9b3c-0808600fa625\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " Jul 28 12:35:44.260648 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.260635 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-trusted-ca-bundle\") pod \"a7f981e6-8e16-4920-9b3c-0808600fa625\" (UID: \"a7f981e6-8e16-4920-9b3c-0808600fa625\") " Jul 28 12:35:44.260908 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.260837 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-console-config" (OuterVolumeSpecName: "console-config") pod "a7f981e6-8e16-4920-9b3c-0808600fa625" (UID: "a7f981e6-8e16-4920-9b3c-0808600fa625"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:35:44.261084 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.261046 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-service-ca" (OuterVolumeSpecName: "service-ca") pod "a7f981e6-8e16-4920-9b3c-0808600fa625" (UID: "a7f981e6-8e16-4920-9b3c-0808600fa625"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:35:44.261084 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.261056 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a7f981e6-8e16-4920-9b3c-0808600fa625" (UID: "a7f981e6-8e16-4920-9b3c-0808600fa625"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:35:44.261265 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.261108 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a7f981e6-8e16-4920-9b3c-0808600fa625" (UID: "a7f981e6-8e16-4920-9b3c-0808600fa625"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:35:44.263036 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.263009 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f981e6-8e16-4920-9b3c-0808600fa625-kube-api-access-cr2hh" (OuterVolumeSpecName: "kube-api-access-cr2hh") pod "a7f981e6-8e16-4920-9b3c-0808600fa625" (UID: "a7f981e6-8e16-4920-9b3c-0808600fa625"). InnerVolumeSpecName "kube-api-access-cr2hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:35:44.263149 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.263130 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f981e6-8e16-4920-9b3c-0808600fa625-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a7f981e6-8e16-4920-9b3c-0808600fa625" (UID: "a7f981e6-8e16-4920-9b3c-0808600fa625"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:35:44.263206 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.263147 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f981e6-8e16-4920-9b3c-0808600fa625-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a7f981e6-8e16-4920-9b3c-0808600fa625" (UID: "a7f981e6-8e16-4920-9b3c-0808600fa625"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:35:44.361672 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.361636 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr2hh\" (UniqueName: \"kubernetes.io/projected/a7f981e6-8e16-4920-9b3c-0808600fa625-kube-api-access-cr2hh\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:44.361672 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.361672 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-trusted-ca-bundle\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:44.361883 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.361686 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-console-config\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:44.361883 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.361700 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7f981e6-8e16-4920-9b3c-0808600fa625-console-oauth-config\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:44.361883 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.361715 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7f981e6-8e16-4920-9b3c-0808600fa625-console-serving-cert\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:44.361883 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.361727 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:44.361883 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.361739 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7f981e6-8e16-4920-9b3c-0808600fa625-oauth-serving-cert\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:44.467553 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.467517 2573 generic.go:356] "Generic (PLEG): container finished" podID="a7f981e6-8e16-4920-9b3c-0808600fa625" containerID="6ccadc6956fb3fc05179296bc8080b9dbd5fc30b02169d241a05bd6f06d3fecf" exitCode=0 Jul 28 12:35:44.467718 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.467565 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bd755f5-q7fbx" Jul 28 12:35:44.467718 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.467589 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bd755f5-q7fbx" event={"ID":"a7f981e6-8e16-4920-9b3c-0808600fa625","Type":"ContainerDied","Data":"6ccadc6956fb3fc05179296bc8080b9dbd5fc30b02169d241a05bd6f06d3fecf"} Jul 28 12:35:44.467718 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.467636 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bd755f5-q7fbx" event={"ID":"a7f981e6-8e16-4920-9b3c-0808600fa625","Type":"ContainerDied","Data":"25ebd461f0f66b223ee9ccf37cd930ba5aaf1cc21f44827dc5b9f7c22108370c"} Jul 28 12:35:44.467718 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.467656 2573 scope.go:117] "RemoveContainer" containerID="6ccadc6956fb3fc05179296bc8080b9dbd5fc30b02169d241a05bd6f06d3fecf" Jul 28 12:35:44.476183 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.476167 2573 scope.go:117] "RemoveContainer" containerID="6ccadc6956fb3fc05179296bc8080b9dbd5fc30b02169d241a05bd6f06d3fecf" Jul 28 12:35:44.476411 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:35:44.476396 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ccadc6956fb3fc05179296bc8080b9dbd5fc30b02169d241a05bd6f06d3fecf\": container with ID starting with 6ccadc6956fb3fc05179296bc8080b9dbd5fc30b02169d241a05bd6f06d3fecf not found: ID does not exist" containerID="6ccadc6956fb3fc05179296bc8080b9dbd5fc30b02169d241a05bd6f06d3fecf" Jul 28 12:35:44.476471 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.476419 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ccadc6956fb3fc05179296bc8080b9dbd5fc30b02169d241a05bd6f06d3fecf"} err="failed to get container status \"6ccadc6956fb3fc05179296bc8080b9dbd5fc30b02169d241a05bd6f06d3fecf\": rpc error: code = NotFound desc = could not find container \"6ccadc6956fb3fc05179296bc8080b9dbd5fc30b02169d241a05bd6f06d3fecf\": container with ID starting with 6ccadc6956fb3fc05179296bc8080b9dbd5fc30b02169d241a05bd6f06d3fecf not found: ID does not exist" Jul 28 12:35:44.488880 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.488857 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74bd755f5-q7fbx"] Jul 28 12:35:44.492200 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:44.492179 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74bd755f5-q7fbx"] Jul 28 12:35:45.267767 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:45.267734 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7f981e6-8e16-4920-9b3c-0808600fa625" path="/var/lib/kubelet/pods/a7f981e6-8e16-4920-9b3c-0808600fa625/volumes" Jul 28 12:35:54.474315 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.474253 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d65b8bb-w5w5w" podUID="dcd08263-e787-44d6-b106-0603f93f54eb" containerName="console" containerID="cri-o://be39b5002461eb1b066f9ef0c4ea1d9600244a9a35937be43dc06e354416e624" gracePeriod=15 Jul 28 12:35:54.723577 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.723556 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:35:54.848546 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.848485 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-oauth-serving-cert\") pod \"dcd08263-e787-44d6-b106-0603f93f54eb\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " Jul 28 12:35:54.848546 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.848513 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnddl\" (UniqueName: \"kubernetes.io/projected/dcd08263-e787-44d6-b106-0603f93f54eb-kube-api-access-bnddl\") pod \"dcd08263-e787-44d6-b106-0603f93f54eb\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " Jul 28 12:35:54.848546 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.848545 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-console-config\") pod \"dcd08263-e787-44d6-b106-0603f93f54eb\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " Jul 28 12:35:54.848783 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.848573 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcd08263-e787-44d6-b106-0603f93f54eb-console-oauth-config\") pod \"dcd08263-e787-44d6-b106-0603f93f54eb\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " Jul 28 12:35:54.848783 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.848610 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-service-ca\") pod \"dcd08263-e787-44d6-b106-0603f93f54eb\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " Jul 28 12:35:54.848783 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.848630 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-trusted-ca-bundle\") pod \"dcd08263-e787-44d6-b106-0603f93f54eb\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " Jul 28 12:35:54.848783 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.848657 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcd08263-e787-44d6-b106-0603f93f54eb-console-serving-cert\") pod \"dcd08263-e787-44d6-b106-0603f93f54eb\" (UID: \"dcd08263-e787-44d6-b106-0603f93f54eb\") " Jul 28 12:35:54.848984 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.848947 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dcd08263-e787-44d6-b106-0603f93f54eb" (UID: "dcd08263-e787-44d6-b106-0603f93f54eb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:35:54.849036 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.848986 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-service-ca" (OuterVolumeSpecName: "service-ca") pod "dcd08263-e787-44d6-b106-0603f93f54eb" (UID: "dcd08263-e787-44d6-b106-0603f93f54eb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:35:54.849036 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.848996 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-console-config" (OuterVolumeSpecName: "console-config") pod "dcd08263-e787-44d6-b106-0603f93f54eb" (UID: "dcd08263-e787-44d6-b106-0603f93f54eb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:35:54.849127 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.849057 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dcd08263-e787-44d6-b106-0603f93f54eb" (UID: "dcd08263-e787-44d6-b106-0603f93f54eb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:35:54.850892 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.850861 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd08263-e787-44d6-b106-0603f93f54eb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dcd08263-e787-44d6-b106-0603f93f54eb" (UID: "dcd08263-e787-44d6-b106-0603f93f54eb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:35:54.850997 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.850919 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd08263-e787-44d6-b106-0603f93f54eb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dcd08263-e787-44d6-b106-0603f93f54eb" (UID: "dcd08263-e787-44d6-b106-0603f93f54eb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:35:54.850997 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.850926 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd08263-e787-44d6-b106-0603f93f54eb-kube-api-access-bnddl" (OuterVolumeSpecName: "kube-api-access-bnddl") pod "dcd08263-e787-44d6-b106-0603f93f54eb" (UID: "dcd08263-e787-44d6-b106-0603f93f54eb"). InnerVolumeSpecName "kube-api-access-bnddl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:35:54.949384 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.949329 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:54.949384 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.949381 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-trusted-ca-bundle\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:54.949384 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.949389 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcd08263-e787-44d6-b106-0603f93f54eb-console-serving-cert\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:54.949594 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.949398 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-oauth-serving-cert\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:54.949594 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.949408 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnddl\" (UniqueName: \"kubernetes.io/projected/dcd08263-e787-44d6-b106-0603f93f54eb-kube-api-access-bnddl\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:54.949594 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.949416 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcd08263-e787-44d6-b106-0603f93f54eb-console-config\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:54.949594 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:54.949424 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcd08263-e787-44d6-b106-0603f93f54eb-console-oauth-config\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:35:55.509489 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:55.509424 2573 generic.go:356] "Generic (PLEG): container finished" podID="dcd08263-e787-44d6-b106-0603f93f54eb" containerID="be39b5002461eb1b066f9ef0c4ea1d9600244a9a35937be43dc06e354416e624" exitCode=0 Jul 28 12:35:55.509941 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:55.509503 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d65b8bb-w5w5w" event={"ID":"dcd08263-e787-44d6-b106-0603f93f54eb","Type":"ContainerDied","Data":"be39b5002461eb1b066f9ef0c4ea1d9600244a9a35937be43dc06e354416e624"} Jul 28 12:35:55.509941 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:55.509548 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d65b8bb-w5w5w" event={"ID":"dcd08263-e787-44d6-b106-0603f93f54eb","Type":"ContainerDied","Data":"2e93186cc504eea42f83a3726dfe315a85239e6608ba8f3adc7463dc89e09bba"} Jul 28 12:35:55.509941 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:55.509577 2573 scope.go:117] "RemoveContainer" containerID="be39b5002461eb1b066f9ef0c4ea1d9600244a9a35937be43dc06e354416e624" Jul 28 12:35:55.509941 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:55.509512 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d65b8bb-w5w5w" Jul 28 12:35:55.517601 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:55.517585 2573 scope.go:117] "RemoveContainer" containerID="be39b5002461eb1b066f9ef0c4ea1d9600244a9a35937be43dc06e354416e624" Jul 28 12:35:55.517870 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:35:55.517852 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be39b5002461eb1b066f9ef0c4ea1d9600244a9a35937be43dc06e354416e624\": container with ID starting with be39b5002461eb1b066f9ef0c4ea1d9600244a9a35937be43dc06e354416e624 not found: ID does not exist" containerID="be39b5002461eb1b066f9ef0c4ea1d9600244a9a35937be43dc06e354416e624" Jul 28 12:35:55.517925 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:55.517878 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be39b5002461eb1b066f9ef0c4ea1d9600244a9a35937be43dc06e354416e624"} err="failed to get container status \"be39b5002461eb1b066f9ef0c4ea1d9600244a9a35937be43dc06e354416e624\": rpc error: code = NotFound desc = could not find container \"be39b5002461eb1b066f9ef0c4ea1d9600244a9a35937be43dc06e354416e624\": container with ID starting with be39b5002461eb1b066f9ef0c4ea1d9600244a9a35937be43dc06e354416e624 not found: ID does not exist" Jul 28 12:35:55.528574 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:55.528530 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d65b8bb-w5w5w"] Jul 28 12:35:55.531760 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:55.531741 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d65b8bb-w5w5w"] Jul 28 12:35:57.266542 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:35:57.266509 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcd08263-e787-44d6-b106-0603f93f54eb" path="/var/lib/kubelet/pods/dcd08263-e787-44d6-b106-0603f93f54eb/volumes" Jul 28 12:36:29.187837 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:29.187801 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:36:29.190152 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:29.190134 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4868cb1-759e-4f1a-a614-165690edf4a3-metrics-certs\") pod \"network-metrics-daemon-csfdh\" (UID: \"a4868cb1-759e-4f1a-a614-165690edf4a3\") " pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:36:29.265921 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:29.265893 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nm4t9\"" Jul 28 12:36:29.273223 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:29.273204 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-csfdh" Jul 28 12:36:29.399101 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:29.399079 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-csfdh"] Jul 28 12:36:29.400760 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:36:29.400739 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4868cb1_759e_4f1a_a614_165690edf4a3.slice/crio-55d438e2a6b320d93f8ab8386465d31502e6702acee6c43a1818dda2630b0e44 WatchSource:0}: Error finding container 55d438e2a6b320d93f8ab8386465d31502e6702acee6c43a1818dda2630b0e44: Status 404 returned error can't find the container with id 55d438e2a6b320d93f8ab8386465d31502e6702acee6c43a1818dda2630b0e44 Jul 28 12:36:29.402912 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:29.402893 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 12:36:29.639712 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:29.639685 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-csfdh" event={"ID":"a4868cb1-759e-4f1a-a614-165690edf4a3","Type":"ContainerStarted","Data":"55d438e2a6b320d93f8ab8386465d31502e6702acee6c43a1818dda2630b0e44"} Jul 28 12:36:30.644696 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:30.644661 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-csfdh" event={"ID":"a4868cb1-759e-4f1a-a614-165690edf4a3","Type":"ContainerStarted","Data":"040fcf2e342c0ade8e41e015c1bd0f6930a81e98400d1bbdd14470e912a70bd4"} Jul 28 12:36:30.645102 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:30.644718 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-csfdh" event={"ID":"a4868cb1-759e-4f1a-a614-165690edf4a3","Type":"ContainerStarted","Data":"69f64fa787b4d1606238ee3a1a9971676bf6ba0cdf351d54b36fb17e9bc4efe6"} Jul 28 12:36:30.661113 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:30.661067 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-csfdh" podStartSLOduration=374.82670287 podStartE2EDuration="6m15.661052518s" totalImagesPullingTime="834.349648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:30.66032218 +0000 UTC m=+375.919612918" watchObservedRunningTime="2026-07-28 12:36:30.661052518 +0000 UTC m=+375.920343257" Jul 28 12:36:37.836798 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.836771 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c4dbddb46-n2z8h"] Jul 28 12:36:37.837250 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.837161 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f981e6-8e16-4920-9b3c-0808600fa625" containerName="console" Jul 28 12:36:37.837250 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.837180 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f981e6-8e16-4920-9b3c-0808600fa625" containerName="console" Jul 28 12:36:37.837250 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.837196 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd08263-e787-44d6-b106-0603f93f54eb" containerName="console" Jul 28 12:36:37.837250 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.837205 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd08263-e787-44d6-b106-0603f93f54eb" containerName="console" Jul 28 12:36:37.837250 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.837225 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d48896cf-9240-4225-8d7c-a68dc89612ce" containerName="console" Jul 28 12:36:37.837250 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.837232 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d48896cf-9240-4225-8d7c-a68dc89612ce" containerName="console" Jul 28 12:36:37.837591 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.837319 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd08263-e787-44d6-b106-0603f93f54eb" containerName="console" Jul 28 12:36:37.837591 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.837332 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d48896cf-9240-4225-8d7c-a68dc89612ce" containerName="console" Jul 28 12:36:37.837591 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.837345 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f981e6-8e16-4920-9b3c-0808600fa625" containerName="console" Jul 28 12:36:37.840236 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.840214 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:37.854160 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.854136 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4dbddb46-n2z8h"] Jul 28 12:36:37.953051 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.953022 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-trusted-ca-bundle\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:37.953220 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.953072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-service-ca\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:37.953220 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.953089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-oauth-serving-cert\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:37.953220 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.953137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-config\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:37.953220 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.953183 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-oauth-config\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:37.953407 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.953233 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmws\" (UniqueName: \"kubernetes.io/projected/16f63b97-a829-4e6d-861b-3d29cbdcd551-kube-api-access-7gmws\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:37.953407 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:37.953260 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-serving-cert\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.053943 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.053904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-service-ca\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.054086 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.053949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-oauth-serving-cert\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.054086 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.053982 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-config\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.054086 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.054002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-oauth-config\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.054086 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.054029 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmws\" (UniqueName: \"kubernetes.io/projected/16f63b97-a829-4e6d-861b-3d29cbdcd551-kube-api-access-7gmws\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.054086 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.054054 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-serving-cert\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.054321 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.054098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-trusted-ca-bundle\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.054696 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.054672 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-service-ca\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.054788 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.054696 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-config\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.054860 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.054771 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-oauth-serving-cert\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.055213 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.055184 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-trusted-ca-bundle\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.056637 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.056615 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-serving-cert\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.056770 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.056749 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-oauth-config\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.063899 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.063877 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmws\" (UniqueName: \"kubernetes.io/projected/16f63b97-a829-4e6d-861b-3d29cbdcd551-kube-api-access-7gmws\") pod \"console-6c4dbddb46-n2z8h\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.149457 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.149408 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:38.282057 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.282034 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4dbddb46-n2z8h"] Jul 28 12:36:38.284135 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:36:38.284114 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f63b97_a829_4e6d_861b_3d29cbdcd551.slice/crio-cda0d401943bd5694a212946a54bc663cc36ad93d356d907b30409142e9813b5 WatchSource:0}: Error finding container cda0d401943bd5694a212946a54bc663cc36ad93d356d907b30409142e9813b5: Status 404 returned error can't find the container with id cda0d401943bd5694a212946a54bc663cc36ad93d356d907b30409142e9813b5 Jul 28 12:36:38.677247 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.677219 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4dbddb46-n2z8h" event={"ID":"16f63b97-a829-4e6d-861b-3d29cbdcd551","Type":"ContainerStarted","Data":"6cdc695cd979ef7918e10793b10062f47b68bbe3d967de20aa62fec1e6d28bb9"} Jul 28 12:36:38.677423 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.677265 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4dbddb46-n2z8h" event={"ID":"16f63b97-a829-4e6d-861b-3d29cbdcd551","Type":"ContainerStarted","Data":"cda0d401943bd5694a212946a54bc663cc36ad93d356d907b30409142e9813b5"} Jul 28 12:36:38.697671 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:38.697625 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c4dbddb46-n2z8h" podStartSLOduration=1.697611535 podStartE2EDuration="1.697611535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:38.695354753 +0000 UTC m=+383.954645495" watchObservedRunningTime="2026-07-28 12:36:38.697611535 +0000 UTC m=+383.956902276" Jul 28 12:36:48.150539 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:48.150498 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:48.150912 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:48.150570 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:48.155426 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:48.155406 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:48.716173 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:48.716147 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:36:48.764376 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:36:48.764345 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f7787dbd8-7h5dx"] Jul 28 12:37:13.784450 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:13.784388 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f7787dbd8-7h5dx" podUID="000c6e16-dd51-4e8e-8689-351387700681" containerName="console" containerID="cri-o://1474ea9b171a68dc5f4556de8fdeda29f0375b98617959af16ff6d3ebc9d4882" gracePeriod=15 Jul 28 12:37:14.032386 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.032357 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:37:14.043018 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.042949 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-console-config\") pod \"000c6e16-dd51-4e8e-8689-351387700681\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " Jul 28 12:37:14.043018 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.042985 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-oauth-serving-cert\") pod \"000c6e16-dd51-4e8e-8689-351387700681\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " Jul 28 12:37:14.043018 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.043000 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-trusted-ca-bundle\") pod \"000c6e16-dd51-4e8e-8689-351387700681\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " Jul 28 12:37:14.043239 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.043037 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g58p\" (UniqueName: \"kubernetes.io/projected/000c6e16-dd51-4e8e-8689-351387700681-kube-api-access-2g58p\") pod \"000c6e16-dd51-4e8e-8689-351387700681\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " Jul 28 12:37:14.043239 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.043073 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/000c6e16-dd51-4e8e-8689-351387700681-console-oauth-config\") pod \"000c6e16-dd51-4e8e-8689-351387700681\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " Jul 28 12:37:14.043239 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.043137 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/000c6e16-dd51-4e8e-8689-351387700681-console-serving-cert\") pod \"000c6e16-dd51-4e8e-8689-351387700681\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " Jul 28 12:37:14.043239 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.043172 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-service-ca\") pod \"000c6e16-dd51-4e8e-8689-351387700681\" (UID: \"000c6e16-dd51-4e8e-8689-351387700681\") " Jul 28 12:37:14.043462 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.043355 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-console-config" (OuterVolumeSpecName: "console-config") pod "000c6e16-dd51-4e8e-8689-351387700681" (UID: "000c6e16-dd51-4e8e-8689-351387700681"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:37:14.043518 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.043427 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "000c6e16-dd51-4e8e-8689-351387700681" (UID: "000c6e16-dd51-4e8e-8689-351387700681"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:37:14.043577 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.043530 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "000c6e16-dd51-4e8e-8689-351387700681" (UID: "000c6e16-dd51-4e8e-8689-351387700681"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:37:14.043623 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.043601 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-service-ca" (OuterVolumeSpecName: "service-ca") pod "000c6e16-dd51-4e8e-8689-351387700681" (UID: "000c6e16-dd51-4e8e-8689-351387700681"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:37:14.045374 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.045350 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000c6e16-dd51-4e8e-8689-351387700681-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "000c6e16-dd51-4e8e-8689-351387700681" (UID: "000c6e16-dd51-4e8e-8689-351387700681"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:37:14.045490 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.045379 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000c6e16-dd51-4e8e-8689-351387700681-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "000c6e16-dd51-4e8e-8689-351387700681" (UID: "000c6e16-dd51-4e8e-8689-351387700681"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:37:14.045536 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.045501 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000c6e16-dd51-4e8e-8689-351387700681-kube-api-access-2g58p" (OuterVolumeSpecName: "kube-api-access-2g58p") pod "000c6e16-dd51-4e8e-8689-351387700681" (UID: "000c6e16-dd51-4e8e-8689-351387700681"). InnerVolumeSpecName "kube-api-access-2g58p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:37:14.143981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.143935 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g58p\" (UniqueName: \"kubernetes.io/projected/000c6e16-dd51-4e8e-8689-351387700681-kube-api-access-2g58p\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:37:14.143981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.143975 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/000c6e16-dd51-4e8e-8689-351387700681-console-oauth-config\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:37:14.143981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.143984 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/000c6e16-dd51-4e8e-8689-351387700681-console-serving-cert\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:37:14.143981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.143993 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:37:14.143981 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.144001 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-console-config\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:37:14.144271 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.144009 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-oauth-serving-cert\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:37:14.144271 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.144018 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/000c6e16-dd51-4e8e-8689-351387700681-trusted-ca-bundle\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:37:14.813882 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.813850 2573 generic.go:356] "Generic (PLEG): container finished" podID="000c6e16-dd51-4e8e-8689-351387700681" containerID="1474ea9b171a68dc5f4556de8fdeda29f0375b98617959af16ff6d3ebc9d4882" exitCode=0 Jul 28 12:37:14.814347 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.813907 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7787dbd8-7h5dx" Jul 28 12:37:14.814347 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.813947 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7787dbd8-7h5dx" event={"ID":"000c6e16-dd51-4e8e-8689-351387700681","Type":"ContainerDied","Data":"1474ea9b171a68dc5f4556de8fdeda29f0375b98617959af16ff6d3ebc9d4882"} Jul 28 12:37:14.814347 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.814003 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7787dbd8-7h5dx" event={"ID":"000c6e16-dd51-4e8e-8689-351387700681","Type":"ContainerDied","Data":"b021aebbd2165948a32144522fe9703e023758d87bd645828f169309b0ddf418"} Jul 28 12:37:14.814347 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.814029 2573 scope.go:117] "RemoveContainer" containerID="1474ea9b171a68dc5f4556de8fdeda29f0375b98617959af16ff6d3ebc9d4882" Jul 28 12:37:14.823193 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.823175 2573 scope.go:117] "RemoveContainer" containerID="1474ea9b171a68dc5f4556de8fdeda29f0375b98617959af16ff6d3ebc9d4882" Jul 28 12:37:14.823459 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:37:14.823415 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1474ea9b171a68dc5f4556de8fdeda29f0375b98617959af16ff6d3ebc9d4882\": container with ID starting with 1474ea9b171a68dc5f4556de8fdeda29f0375b98617959af16ff6d3ebc9d4882 not found: ID does not exist" containerID="1474ea9b171a68dc5f4556de8fdeda29f0375b98617959af16ff6d3ebc9d4882" Jul 28 12:37:14.823542 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.823465 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1474ea9b171a68dc5f4556de8fdeda29f0375b98617959af16ff6d3ebc9d4882"} err="failed to get container status \"1474ea9b171a68dc5f4556de8fdeda29f0375b98617959af16ff6d3ebc9d4882\": rpc error: code = NotFound desc = could not find container \"1474ea9b171a68dc5f4556de8fdeda29f0375b98617959af16ff6d3ebc9d4882\": container with ID starting with 1474ea9b171a68dc5f4556de8fdeda29f0375b98617959af16ff6d3ebc9d4882 not found: ID does not exist" Jul 28 12:37:14.835220 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.835197 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f7787dbd8-7h5dx"] Jul 28 12:37:14.839861 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:14.839815 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f7787dbd8-7h5dx"] Jul 28 12:37:15.265751 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:37:15.265722 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="000c6e16-dd51-4e8e-8689-351387700681" path="/var/lib/kubelet/pods/000c6e16-dd51-4e8e-8689-351387700681/volumes" Jul 28 12:38:46.217307 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.217264 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4"] Jul 28 12:38:46.217780 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.217563 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="000c6e16-dd51-4e8e-8689-351387700681" containerName="console" Jul 28 12:38:46.217780 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.217582 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="000c6e16-dd51-4e8e-8689-351387700681" containerName="console" Jul 28 12:38:46.217780 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.217649 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="000c6e16-dd51-4e8e-8689-351387700681" containerName="console" Jul 28 12:38:46.220841 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.220822 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" Jul 28 12:38:46.224015 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.223991 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 12:38:46.224015 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.224009 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 12:38:46.224175 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.223996 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nh8n9\"" Jul 28 12:38:46.247774 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.247737 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4"] Jul 28 12:38:46.315481 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.315412 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-845zz\" (UniqueName: \"kubernetes.io/projected/10a9385d-eaab-4252-8ba5-eac1d429e9e2-kube-api-access-845zz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4\" (UID: \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" Jul 28 12:38:46.315620 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.315485 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10a9385d-eaab-4252-8ba5-eac1d429e9e2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4\" (UID: \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" Jul 28 12:38:46.315620 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.315546 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10a9385d-eaab-4252-8ba5-eac1d429e9e2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4\" (UID: \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" Jul 28 12:38:46.416076 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.416038 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10a9385d-eaab-4252-8ba5-eac1d429e9e2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4\" (UID: \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" Jul 28 12:38:46.416254 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.416087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10a9385d-eaab-4252-8ba5-eac1d429e9e2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4\" (UID: \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" Jul 28 12:38:46.416254 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.416172 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-845zz\" (UniqueName: \"kubernetes.io/projected/10a9385d-eaab-4252-8ba5-eac1d429e9e2-kube-api-access-845zz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4\" (UID: \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" Jul 28 12:38:46.416461 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.416417 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10a9385d-eaab-4252-8ba5-eac1d429e9e2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4\" (UID: \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" Jul 28 12:38:46.416546 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.416487 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10a9385d-eaab-4252-8ba5-eac1d429e9e2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4\" (UID: \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" Jul 28 12:38:46.425393 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.425371 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-845zz\" (UniqueName: \"kubernetes.io/projected/10a9385d-eaab-4252-8ba5-eac1d429e9e2-kube-api-access-845zz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4\" (UID: \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" Jul 28 12:38:46.530325 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.530225 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" Jul 28 12:38:46.657339 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:46.657316 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4"] Jul 28 12:38:46.659254 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:38:46.659220 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a9385d_eaab_4252_8ba5_eac1d429e9e2.slice/crio-3d149055b605df74264bb35bbbc07f192c84ead4c0d19b9130cfee44a7866efd WatchSource:0}: Error finding container 3d149055b605df74264bb35bbbc07f192c84ead4c0d19b9130cfee44a7866efd: Status 404 returned error can't find the container with id 3d149055b605df74264bb35bbbc07f192c84ead4c0d19b9130cfee44a7866efd Jul 28 12:38:47.155092 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:47.155059 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" event={"ID":"10a9385d-eaab-4252-8ba5-eac1d429e9e2","Type":"ContainerStarted","Data":"3d149055b605df74264bb35bbbc07f192c84ead4c0d19b9130cfee44a7866efd"} Jul 28 12:38:54.185266 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:54.185239 2573 generic.go:356] "Generic (PLEG): container finished" podID="10a9385d-eaab-4252-8ba5-eac1d429e9e2" containerID="57c95b04435a6611ccc7c26bca5bd9d0512726f698f7da18f1817689a8528c61" exitCode=0 Jul 28 12:38:54.185266 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:38:54.185272 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" event={"ID":"10a9385d-eaab-4252-8ba5-eac1d429e9e2","Type":"ContainerDied","Data":"57c95b04435a6611ccc7c26bca5bd9d0512726f698f7da18f1817689a8528c61"} Jul 28 12:39:01.214342 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:01.214306 2573 generic.go:356] "Generic (PLEG): container finished" podID="10a9385d-eaab-4252-8ba5-eac1d429e9e2" containerID="2b388e584723e0597455d862a8628cf6edf4badf7413b8eb3ef3f5ee85e17d8a" exitCode=0 Jul 28 12:39:01.214747 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:01.214394 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" event={"ID":"10a9385d-eaab-4252-8ba5-eac1d429e9e2","Type":"ContainerDied","Data":"2b388e584723e0597455d862a8628cf6edf4badf7413b8eb3ef3f5ee85e17d8a"} Jul 28 12:39:08.245194 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:08.245147 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" event={"ID":"10a9385d-eaab-4252-8ba5-eac1d429e9e2","Type":"ContainerStarted","Data":"ed0899957f7c030e024f60ecec15e75ce1e71e5627da99b8ad69bc6b6bef9b42"} Jul 28 12:39:08.264965 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:08.264870 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" podStartSLOduration=2.231897409 podStartE2EDuration="22.264853875s" totalImagesPullingTime="20.032956466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:38:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:39:08.263053171 +0000 UTC m=+533.522343911" watchObservedRunningTime="2026-07-28 12:39:08.264853875 +0000 UTC m=+533.524144636" Jul 28 12:39:09.251258 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:09.251223 2573 generic.go:356] "Generic (PLEG): container finished" podID="10a9385d-eaab-4252-8ba5-eac1d429e9e2" containerID="ed0899957f7c030e024f60ecec15e75ce1e71e5627da99b8ad69bc6b6bef9b42" exitCode=0 Jul 28 12:39:09.251686 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:09.251272 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" event={"ID":"10a9385d-eaab-4252-8ba5-eac1d429e9e2","Type":"ContainerDied","Data":"ed0899957f7c030e024f60ecec15e75ce1e71e5627da99b8ad69bc6b6bef9b42"} Jul 28 12:39:10.399722 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:10.399688 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" Jul 28 12:39:10.503630 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:10.503595 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-845zz\" (UniqueName: \"kubernetes.io/projected/10a9385d-eaab-4252-8ba5-eac1d429e9e2-kube-api-access-845zz\") pod \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\" (UID: \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\") " Jul 28 12:39:10.503822 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:10.503658 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10a9385d-eaab-4252-8ba5-eac1d429e9e2-util\") pod \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\" (UID: \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\") " Jul 28 12:39:10.503822 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:10.503695 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10a9385d-eaab-4252-8ba5-eac1d429e9e2-bundle\") pod \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\" (UID: \"10a9385d-eaab-4252-8ba5-eac1d429e9e2\") " Jul 28 12:39:10.504313 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:10.504288 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a9385d-eaab-4252-8ba5-eac1d429e9e2-bundle" (OuterVolumeSpecName: "bundle") pod "10a9385d-eaab-4252-8ba5-eac1d429e9e2" (UID: "10a9385d-eaab-4252-8ba5-eac1d429e9e2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:39:10.505998 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:10.505945 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a9385d-eaab-4252-8ba5-eac1d429e9e2-kube-api-access-845zz" (OuterVolumeSpecName: "kube-api-access-845zz") pod "10a9385d-eaab-4252-8ba5-eac1d429e9e2" (UID: "10a9385d-eaab-4252-8ba5-eac1d429e9e2"). InnerVolumeSpecName "kube-api-access-845zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:39:10.508023 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:10.507994 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a9385d-eaab-4252-8ba5-eac1d429e9e2-util" (OuterVolumeSpecName: "util") pod "10a9385d-eaab-4252-8ba5-eac1d429e9e2" (UID: "10a9385d-eaab-4252-8ba5-eac1d429e9e2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:39:10.604542 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:10.604508 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10a9385d-eaab-4252-8ba5-eac1d429e9e2-util\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:39:10.604542 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:10.604534 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10a9385d-eaab-4252-8ba5-eac1d429e9e2-bundle\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:39:10.604542 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:10.604545 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-845zz\" (UniqueName: \"kubernetes.io/projected/10a9385d-eaab-4252-8ba5-eac1d429e9e2-kube-api-access-845zz\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:39:11.260933 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:11.260900 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" event={"ID":"10a9385d-eaab-4252-8ba5-eac1d429e9e2","Type":"ContainerDied","Data":"3d149055b605df74264bb35bbbc07f192c84ead4c0d19b9130cfee44a7866efd"} Jul 28 12:39:11.261102 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:11.260943 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675405b9f4" Jul 28 12:39:11.261102 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:11.260952 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d149055b605df74264bb35bbbc07f192c84ead4c0d19b9130cfee44a7866efd" Jul 28 12:39:14.244715 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.244681 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj"] Jul 28 12:39:14.245080 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.244946 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a9385d-eaab-4252-8ba5-eac1d429e9e2" containerName="pull" Jul 28 12:39:14.245080 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.244956 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a9385d-eaab-4252-8ba5-eac1d429e9e2" containerName="pull" Jul 28 12:39:14.245080 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.244976 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a9385d-eaab-4252-8ba5-eac1d429e9e2" containerName="extract" Jul 28 12:39:14.245080 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.244982 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a9385d-eaab-4252-8ba5-eac1d429e9e2" containerName="extract" Jul 28 12:39:14.245080 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.244994 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a9385d-eaab-4252-8ba5-eac1d429e9e2" containerName="util" Jul 28 12:39:14.245080 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.244998 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a9385d-eaab-4252-8ba5-eac1d429e9e2" containerName="util" Jul 28 12:39:14.245080 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.245043 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a9385d-eaab-4252-8ba5-eac1d429e9e2" containerName="extract" Jul 28 12:39:14.246823 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.246809 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj" Jul 28 12:39:14.249655 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.249629 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 28 12:39:14.249655 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.249648 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5qgvv\"" Jul 28 12:39:14.249838 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.249670 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 28 12:39:14.259824 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.259780 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj"] Jul 28 12:39:14.332415 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.332367 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9dq\" (UniqueName: \"kubernetes.io/projected/729fbeaf-c1fe-4bb7-9b22-79919ef9a89a-kube-api-access-zn9dq\") pod \"cert-manager-operator-controller-manager-55c444bf86-bblkj\" (UID: \"729fbeaf-c1fe-4bb7-9b22-79919ef9a89a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj" Jul 28 12:39:14.332415 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.332418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/729fbeaf-c1fe-4bb7-9b22-79919ef9a89a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bblkj\" (UID: \"729fbeaf-c1fe-4bb7-9b22-79919ef9a89a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj" Jul 28 12:39:14.433564 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.433523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9dq\" (UniqueName: \"kubernetes.io/projected/729fbeaf-c1fe-4bb7-9b22-79919ef9a89a-kube-api-access-zn9dq\") pod \"cert-manager-operator-controller-manager-55c444bf86-bblkj\" (UID: \"729fbeaf-c1fe-4bb7-9b22-79919ef9a89a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj" Jul 28 12:39:14.433564 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.433569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/729fbeaf-c1fe-4bb7-9b22-79919ef9a89a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bblkj\" (UID: \"729fbeaf-c1fe-4bb7-9b22-79919ef9a89a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj" Jul 28 12:39:14.433944 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.433929 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/729fbeaf-c1fe-4bb7-9b22-79919ef9a89a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bblkj\" (UID: \"729fbeaf-c1fe-4bb7-9b22-79919ef9a89a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj" Jul 28 12:39:14.442481 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.442429 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9dq\" (UniqueName: \"kubernetes.io/projected/729fbeaf-c1fe-4bb7-9b22-79919ef9a89a-kube-api-access-zn9dq\") pod \"cert-manager-operator-controller-manager-55c444bf86-bblkj\" (UID: \"729fbeaf-c1fe-4bb7-9b22-79919ef9a89a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj" Jul 28 12:39:14.555762 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.555679 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj" Jul 28 12:39:14.689944 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:14.689910 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj"] Jul 28 12:39:14.692186 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:39:14.692160 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729fbeaf_c1fe_4bb7_9b22_79919ef9a89a.slice/crio-ea7bbbb7a098a5ade4508427ac6471f7c70f7f93ada6908ed52983ff8c271e27 WatchSource:0}: Error finding container ea7bbbb7a098a5ade4508427ac6471f7c70f7f93ada6908ed52983ff8c271e27: Status 404 returned error can't find the container with id ea7bbbb7a098a5ade4508427ac6471f7c70f7f93ada6908ed52983ff8c271e27 Jul 28 12:39:15.276823 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:15.276787 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj" event={"ID":"729fbeaf-c1fe-4bb7-9b22-79919ef9a89a","Type":"ContainerStarted","Data":"ea7bbbb7a098a5ade4508427ac6471f7c70f7f93ada6908ed52983ff8c271e27"} Jul 28 12:39:17.290962 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:17.290928 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj" event={"ID":"729fbeaf-c1fe-4bb7-9b22-79919ef9a89a","Type":"ContainerStarted","Data":"5e4c5afebf39d85ce5314fbf5c51e708fb67b8e2db0a3dda17f3c914f6241301"} Jul 28 12:39:17.313357 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:17.313297 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bblkj" podStartSLOduration=1.331929646 podStartE2EDuration="3.313274902s" totalImagesPullingTime="1.981345256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:39:17.312399998 +0000 UTC m=+542.571690740" watchObservedRunningTime="2026-07-28 12:39:17.313274902 +0000 UTC m=+542.572565642" Jul 28 12:39:21.922950 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:21.922916 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr"] Jul 28 12:39:21.925403 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:21.925386 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" Jul 28 12:39:21.928090 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:21.928067 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 12:39:21.928210 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:21.928110 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nh8n9\"" Jul 28 12:39:21.928210 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:21.928131 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 12:39:21.934564 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:21.934544 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr"] Jul 28 12:39:21.992037 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:21.992011 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/762469c8-e192-41f6-a839-443ddb137ae2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr\" (UID: \"762469c8-e192-41f6-a839-443ddb137ae2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" Jul 28 12:39:21.992157 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:21.992096 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/762469c8-e192-41f6-a839-443ddb137ae2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr\" (UID: \"762469c8-e192-41f6-a839-443ddb137ae2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" Jul 28 12:39:21.992157 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:21.992136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chh2l\" (UniqueName: \"kubernetes.io/projected/762469c8-e192-41f6-a839-443ddb137ae2-kube-api-access-chh2l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr\" (UID: \"762469c8-e192-41f6-a839-443ddb137ae2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" Jul 28 12:39:22.092568 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.092548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/762469c8-e192-41f6-a839-443ddb137ae2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr\" (UID: \"762469c8-e192-41f6-a839-443ddb137ae2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" Jul 28 12:39:22.092657 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.092620 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/762469c8-e192-41f6-a839-443ddb137ae2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr\" (UID: \"762469c8-e192-41f6-a839-443ddb137ae2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" Jul 28 12:39:22.092724 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.092665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chh2l\" (UniqueName: \"kubernetes.io/projected/762469c8-e192-41f6-a839-443ddb137ae2-kube-api-access-chh2l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr\" (UID: \"762469c8-e192-41f6-a839-443ddb137ae2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" Jul 28 12:39:22.092910 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.092888 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/762469c8-e192-41f6-a839-443ddb137ae2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr\" (UID: \"762469c8-e192-41f6-a839-443ddb137ae2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" Jul 28 12:39:22.093000 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.092922 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/762469c8-e192-41f6-a839-443ddb137ae2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr\" (UID: \"762469c8-e192-41f6-a839-443ddb137ae2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" Jul 28 12:39:22.100759 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.100733 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chh2l\" (UniqueName: \"kubernetes.io/projected/762469c8-e192-41f6-a839-443ddb137ae2-kube-api-access-chh2l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr\" (UID: \"762469c8-e192-41f6-a839-443ddb137ae2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" Jul 28 12:39:22.234917 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.234855 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" Jul 28 12:39:22.360399 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.360376 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr"] Jul 28 12:39:22.362630 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:39:22.362609 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762469c8_e192_41f6_a839_443ddb137ae2.slice/crio-d7cafc045364056f8bb9b1c87669d903078334030b2939803d5618d2f5c82bea WatchSource:0}: Error finding container d7cafc045364056f8bb9b1c87669d903078334030b2939803d5618d2f5c82bea: Status 404 returned error can't find the container with id d7cafc045364056f8bb9b1c87669d903078334030b2939803d5618d2f5c82bea Jul 28 12:39:22.535169 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.535095 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9"] Jul 28 12:39:22.537160 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.537141 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9" Jul 28 12:39:22.541073 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.541054 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-9pm7q\"" Jul 28 12:39:22.548741 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.548724 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 12:39:22.548873 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.548858 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 12:39:22.562082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.562062 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9"] Jul 28 12:39:22.597751 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.597716 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a930994f-da0e-4cfa-b226-f84d3cdc8937-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4jwc9\" (UID: \"a930994f-da0e-4cfa-b226-f84d3cdc8937\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9" Jul 28 12:39:22.597885 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.597865 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb58n\" (UniqueName: \"kubernetes.io/projected/a930994f-da0e-4cfa-b226-f84d3cdc8937-kube-api-access-sb58n\") pod \"cert-manager-cainjector-559cdd8c87-4jwc9\" (UID: \"a930994f-da0e-4cfa-b226-f84d3cdc8937\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9" Jul 28 12:39:22.698388 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.698354 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb58n\" (UniqueName: \"kubernetes.io/projected/a930994f-da0e-4cfa-b226-f84d3cdc8937-kube-api-access-sb58n\") pod \"cert-manager-cainjector-559cdd8c87-4jwc9\" (UID: \"a930994f-da0e-4cfa-b226-f84d3cdc8937\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9" Jul 28 12:39:22.698547 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.698416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a930994f-da0e-4cfa-b226-f84d3cdc8937-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4jwc9\" (UID: \"a930994f-da0e-4cfa-b226-f84d3cdc8937\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9" Jul 28 12:39:22.706612 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.706586 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a930994f-da0e-4cfa-b226-f84d3cdc8937-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4jwc9\" (UID: \"a930994f-da0e-4cfa-b226-f84d3cdc8937\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9" Jul 28 12:39:22.706745 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.706727 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb58n\" (UniqueName: \"kubernetes.io/projected/a930994f-da0e-4cfa-b226-f84d3cdc8937-kube-api-access-sb58n\") pod \"cert-manager-cainjector-559cdd8c87-4jwc9\" (UID: \"a930994f-da0e-4cfa-b226-f84d3cdc8937\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9" Jul 28 12:39:22.846585 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.846519 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9" Jul 28 12:39:22.969412 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:22.969385 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9"] Jul 28 12:39:22.972089 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:39:22.972060 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda930994f_da0e_4cfa_b226_f84d3cdc8937.slice/crio-697632a76ea3cecfc2c4247d3445c59c165c3feb524dcbc93a7a14a311e7e25e WatchSource:0}: Error finding container 697632a76ea3cecfc2c4247d3445c59c165c3feb524dcbc93a7a14a311e7e25e: Status 404 returned error can't find the container with id 697632a76ea3cecfc2c4247d3445c59c165c3feb524dcbc93a7a14a311e7e25e Jul 28 12:39:23.314982 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:23.314943 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9" event={"ID":"a930994f-da0e-4cfa-b226-f84d3cdc8937","Type":"ContainerStarted","Data":"697632a76ea3cecfc2c4247d3445c59c165c3feb524dcbc93a7a14a311e7e25e"} Jul 28 12:39:23.316394 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:23.316367 2573 generic.go:356] "Generic (PLEG): container finished" podID="762469c8-e192-41f6-a839-443ddb137ae2" containerID="ac8d91fb58c44c9ec22b22307f08a0c3a77fd61fced7d9b92dd2a59fbb21cc28" exitCode=0 Jul 28 12:39:23.316521 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:23.316412 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" event={"ID":"762469c8-e192-41f6-a839-443ddb137ae2","Type":"ContainerDied","Data":"ac8d91fb58c44c9ec22b22307f08a0c3a77fd61fced7d9b92dd2a59fbb21cc28"} Jul 28 12:39:23.316521 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:23.316464 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" event={"ID":"762469c8-e192-41f6-a839-443ddb137ae2","Type":"ContainerStarted","Data":"d7cafc045364056f8bb9b1c87669d903078334030b2939803d5618d2f5c82bea"} Jul 28 12:39:25.544477 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:25.544425 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9vj55"] Jul 28 12:39:25.546593 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:25.546578 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9vj55" Jul 28 12:39:25.550639 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:25.550616 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-sq87r\"" Jul 28 12:39:25.556795 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:25.556770 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9vj55"] Jul 28 12:39:25.625171 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:25.625141 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-444vm\" (UniqueName: \"kubernetes.io/projected/283e401b-453f-45d6-a703-e566205bdebb-kube-api-access-444vm\") pod \"cert-manager-webhook-6486ffd997-9vj55\" (UID: \"283e401b-453f-45d6-a703-e566205bdebb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9vj55" Jul 28 12:39:25.625335 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:25.625195 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/283e401b-453f-45d6-a703-e566205bdebb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9vj55\" (UID: \"283e401b-453f-45d6-a703-e566205bdebb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9vj55" Jul 28 12:39:25.726488 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:25.726453 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-444vm\" (UniqueName: \"kubernetes.io/projected/283e401b-453f-45d6-a703-e566205bdebb-kube-api-access-444vm\") pod \"cert-manager-webhook-6486ffd997-9vj55\" (UID: \"283e401b-453f-45d6-a703-e566205bdebb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9vj55" Jul 28 12:39:25.726651 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:25.726510 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/283e401b-453f-45d6-a703-e566205bdebb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9vj55\" (UID: \"283e401b-453f-45d6-a703-e566205bdebb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9vj55" Jul 28 12:39:25.740143 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:25.740105 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/283e401b-453f-45d6-a703-e566205bdebb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9vj55\" (UID: \"283e401b-453f-45d6-a703-e566205bdebb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9vj55" Jul 28 12:39:25.740383 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:25.740353 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-444vm\" (UniqueName: \"kubernetes.io/projected/283e401b-453f-45d6-a703-e566205bdebb-kube-api-access-444vm\") pod \"cert-manager-webhook-6486ffd997-9vj55\" (UID: \"283e401b-453f-45d6-a703-e566205bdebb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9vj55" Jul 28 12:39:25.857313 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:25.857246 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9vj55" Jul 28 12:39:26.417962 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:26.417937 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9vj55"] Jul 28 12:39:26.419371 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:39:26.419347 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod283e401b_453f_45d6_a703_e566205bdebb.slice/crio-2a90f9b9375864a2e0a36ed8bc6f6f1520a7ee5c4a1617edaea489e9ace8ceda WatchSource:0}: Error finding container 2a90f9b9375864a2e0a36ed8bc6f6f1520a7ee5c4a1617edaea489e9ace8ceda: Status 404 returned error can't find the container with id 2a90f9b9375864a2e0a36ed8bc6f6f1520a7ee5c4a1617edaea489e9ace8ceda Jul 28 12:39:27.338988 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:27.338947 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9vj55" event={"ID":"283e401b-453f-45d6-a703-e566205bdebb","Type":"ContainerStarted","Data":"e655e434cc72e43281cd5721e55817a28e0c719b71ba2c168935aca6731c7151"} Jul 28 12:39:27.339418 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:27.338994 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9vj55" event={"ID":"283e401b-453f-45d6-a703-e566205bdebb","Type":"ContainerStarted","Data":"2a90f9b9375864a2e0a36ed8bc6f6f1520a7ee5c4a1617edaea489e9ace8ceda"} Jul 28 12:39:27.339418 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:27.339019 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9vj55" Jul 28 12:39:27.340517 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:27.340495 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9" event={"ID":"a930994f-da0e-4cfa-b226-f84d3cdc8937","Type":"ContainerStarted","Data":"5795f8e6c680dbc9333d56d5354ae129f611909a5df98ffc13bf0a6cfb6c331a"} Jul 28 12:39:27.342366 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:27.342345 2573 generic.go:356] "Generic (PLEG): container finished" podID="762469c8-e192-41f6-a839-443ddb137ae2" containerID="b4abd19babde50ad3588c6212abfbb4c1951fa82e23824a285e36e9c6d780025" exitCode=0 Jul 28 12:39:27.342470 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:27.342371 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" event={"ID":"762469c8-e192-41f6-a839-443ddb137ae2","Type":"ContainerDied","Data":"b4abd19babde50ad3588c6212abfbb4c1951fa82e23824a285e36e9c6d780025"} Jul 28 12:39:27.355098 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:27.355050 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-9vj55" podStartSLOduration=2.355034281 podStartE2EDuration="2.355034281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:39:27.354752665 +0000 UTC m=+552.614043408" watchObservedRunningTime="2026-07-28 12:39:27.355034281 +0000 UTC m=+552.614325022" Jul 28 12:39:27.388889 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:27.388838 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4jwc9" podStartSLOduration=2.074377251 podStartE2EDuration="5.38881966s" totalImagesPullingTime="3.314442409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:39:27.38712297 +0000 UTC m=+552.646413710" watchObservedRunningTime="2026-07-28 12:39:27.38881966 +0000 UTC m=+552.648110404" Jul 28 12:39:28.348429 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:28.348395 2573 generic.go:356] "Generic (PLEG): container finished" podID="762469c8-e192-41f6-a839-443ddb137ae2" containerID="f4c284d128f2a1a62b1be750fb218eb414b83758844db1ea823b4b3690b50f30" exitCode=0 Jul 28 12:39:28.348880 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:28.348478 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" event={"ID":"762469c8-e192-41f6-a839-443ddb137ae2","Type":"ContainerDied","Data":"f4c284d128f2a1a62b1be750fb218eb414b83758844db1ea823b4b3690b50f30"} Jul 28 12:39:29.505012 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:29.504988 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" Jul 28 12:39:29.558162 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:29.558136 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/762469c8-e192-41f6-a839-443ddb137ae2-bundle\") pod \"762469c8-e192-41f6-a839-443ddb137ae2\" (UID: \"762469c8-e192-41f6-a839-443ddb137ae2\") " Jul 28 12:39:29.558162 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:29.558166 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chh2l\" (UniqueName: \"kubernetes.io/projected/762469c8-e192-41f6-a839-443ddb137ae2-kube-api-access-chh2l\") pod \"762469c8-e192-41f6-a839-443ddb137ae2\" (UID: \"762469c8-e192-41f6-a839-443ddb137ae2\") " Jul 28 12:39:29.558317 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:29.558189 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/762469c8-e192-41f6-a839-443ddb137ae2-util\") pod \"762469c8-e192-41f6-a839-443ddb137ae2\" (UID: \"762469c8-e192-41f6-a839-443ddb137ae2\") " Jul 28 12:39:29.558594 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:29.558569 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762469c8-e192-41f6-a839-443ddb137ae2-bundle" (OuterVolumeSpecName: "bundle") pod "762469c8-e192-41f6-a839-443ddb137ae2" (UID: "762469c8-e192-41f6-a839-443ddb137ae2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:39:29.560287 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:29.560263 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762469c8-e192-41f6-a839-443ddb137ae2-kube-api-access-chh2l" (OuterVolumeSpecName: "kube-api-access-chh2l") pod "762469c8-e192-41f6-a839-443ddb137ae2" (UID: "762469c8-e192-41f6-a839-443ddb137ae2"). InnerVolumeSpecName "kube-api-access-chh2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:39:29.562382 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:29.562343 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762469c8-e192-41f6-a839-443ddb137ae2-util" (OuterVolumeSpecName: "util") pod "762469c8-e192-41f6-a839-443ddb137ae2" (UID: "762469c8-e192-41f6-a839-443ddb137ae2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:39:29.659513 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:29.659490 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/762469c8-e192-41f6-a839-443ddb137ae2-bundle\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:39:29.659513 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:29.659511 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chh2l\" (UniqueName: \"kubernetes.io/projected/762469c8-e192-41f6-a839-443ddb137ae2-kube-api-access-chh2l\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:39:29.659629 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:29.659521 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/762469c8-e192-41f6-a839-443ddb137ae2-util\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:39:30.359019 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:30.358990 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" Jul 28 12:39:30.359019 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:30.359010 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2p2pr" event={"ID":"762469c8-e192-41f6-a839-443ddb137ae2","Type":"ContainerDied","Data":"d7cafc045364056f8bb9b1c87669d903078334030b2939803d5618d2f5c82bea"} Jul 28 12:39:30.359215 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:30.359051 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7cafc045364056f8bb9b1c87669d903078334030b2939803d5618d2f5c82bea" Jul 28 12:39:33.351196 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:33.351158 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9vj55" Jul 28 12:39:42.224853 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.224815 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d"] Jul 28 12:39:42.225288 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.225152 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762469c8-e192-41f6-a839-443ddb137ae2" containerName="extract" Jul 28 12:39:42.225288 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.225164 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="762469c8-e192-41f6-a839-443ddb137ae2" containerName="extract" Jul 28 12:39:42.225288 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.225175 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762469c8-e192-41f6-a839-443ddb137ae2" containerName="pull" Jul 28 12:39:42.225288 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.225180 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="762469c8-e192-41f6-a839-443ddb137ae2" containerName="pull" Jul 28 12:39:42.225288 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.225189 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762469c8-e192-41f6-a839-443ddb137ae2" containerName="util" Jul 28 12:39:42.225288 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.225194 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="762469c8-e192-41f6-a839-443ddb137ae2" containerName="util" Jul 28 12:39:42.225288 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.225271 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="762469c8-e192-41f6-a839-443ddb137ae2" containerName="extract" Jul 28 12:39:42.257204 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.257178 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d"] Jul 28 12:39:42.257340 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.257272 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" Jul 28 12:39:42.260107 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.260090 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nh8n9\"" Jul 28 12:39:42.260221 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.260128 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 12:39:42.260221 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.260135 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 12:39:42.357535 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.357512 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhzfp\" (UniqueName: \"kubernetes.io/projected/46e5b3d1-37a6-48fa-9d76-170d56582163-kube-api-access-jhzfp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d\" (UID: \"46e5b3d1-37a6-48fa-9d76-170d56582163\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" Jul 28 12:39:42.357658 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.357545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46e5b3d1-37a6-48fa-9d76-170d56582163-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d\" (UID: \"46e5b3d1-37a6-48fa-9d76-170d56582163\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" Jul 28 12:39:42.357658 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.357617 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46e5b3d1-37a6-48fa-9d76-170d56582163-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d\" (UID: \"46e5b3d1-37a6-48fa-9d76-170d56582163\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" Jul 28 12:39:42.458072 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.458049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46e5b3d1-37a6-48fa-9d76-170d56582163-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d\" (UID: \"46e5b3d1-37a6-48fa-9d76-170d56582163\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" Jul 28 12:39:42.458176 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.458103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhzfp\" (UniqueName: \"kubernetes.io/projected/46e5b3d1-37a6-48fa-9d76-170d56582163-kube-api-access-jhzfp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d\" (UID: \"46e5b3d1-37a6-48fa-9d76-170d56582163\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" Jul 28 12:39:42.458176 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.458125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46e5b3d1-37a6-48fa-9d76-170d56582163-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d\" (UID: \"46e5b3d1-37a6-48fa-9d76-170d56582163\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" Jul 28 12:39:42.458455 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.458422 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46e5b3d1-37a6-48fa-9d76-170d56582163-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d\" (UID: \"46e5b3d1-37a6-48fa-9d76-170d56582163\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" Jul 28 12:39:42.458493 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.458475 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46e5b3d1-37a6-48fa-9d76-170d56582163-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d\" (UID: \"46e5b3d1-37a6-48fa-9d76-170d56582163\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" Jul 28 12:39:42.468997 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.468971 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhzfp\" (UniqueName: \"kubernetes.io/projected/46e5b3d1-37a6-48fa-9d76-170d56582163-kube-api-access-jhzfp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d\" (UID: \"46e5b3d1-37a6-48fa-9d76-170d56582163\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" Jul 28 12:39:42.566484 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.566395 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" Jul 28 12:39:42.705713 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:42.705691 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d"] Jul 28 12:39:42.707964 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:39:42.707936 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e5b3d1_37a6_48fa_9d76_170d56582163.slice/crio-8401984e6ad72f8e5ff98ae222551db45ceb2dea49905e8fbff01a26395296e6 WatchSource:0}: Error finding container 8401984e6ad72f8e5ff98ae222551db45ceb2dea49905e8fbff01a26395296e6: Status 404 returned error can't find the container with id 8401984e6ad72f8e5ff98ae222551db45ceb2dea49905e8fbff01a26395296e6 Jul 28 12:39:43.414382 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:43.414349 2573 generic.go:356] "Generic (PLEG): container finished" podID="46e5b3d1-37a6-48fa-9d76-170d56582163" containerID="68347e010a018fece371b1b1c60b5bc4e4f7176be23fbcf4f070eb5d55dd2e73" exitCode=0 Jul 28 12:39:43.414817 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:43.414424 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" event={"ID":"46e5b3d1-37a6-48fa-9d76-170d56582163","Type":"ContainerDied","Data":"68347e010a018fece371b1b1c60b5bc4e4f7176be23fbcf4f070eb5d55dd2e73"} Jul 28 12:39:43.414817 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:43.414481 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" event={"ID":"46e5b3d1-37a6-48fa-9d76-170d56582163","Type":"ContainerStarted","Data":"8401984e6ad72f8e5ff98ae222551db45ceb2dea49905e8fbff01a26395296e6"} Jul 28 12:39:44.420892 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:44.420859 2573 generic.go:356] "Generic (PLEG): container finished" podID="46e5b3d1-37a6-48fa-9d76-170d56582163" containerID="502d21084487b20014db649d3064954d659db4eecbecca6d9fa1ca7869e56cd8" exitCode=0 Jul 28 12:39:44.421297 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:44.420948 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" event={"ID":"46e5b3d1-37a6-48fa-9d76-170d56582163","Type":"ContainerDied","Data":"502d21084487b20014db649d3064954d659db4eecbecca6d9fa1ca7869e56cd8"} Jul 28 12:39:45.426500 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:45.426472 2573 generic.go:356] "Generic (PLEG): container finished" podID="46e5b3d1-37a6-48fa-9d76-170d56582163" containerID="8d269cd46513f565600c7db02ee0cf4b3c9bcc3818d1b20219fb802c937fd692" exitCode=0 Jul 28 12:39:45.426872 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:45.426552 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" event={"ID":"46e5b3d1-37a6-48fa-9d76-170d56582163","Type":"ContainerDied","Data":"8d269cd46513f565600c7db02ee0cf4b3c9bcc3818d1b20219fb802c937fd692"} Jul 28 12:39:46.552481 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:46.552458 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" Jul 28 12:39:46.691964 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:46.691894 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46e5b3d1-37a6-48fa-9d76-170d56582163-util\") pod \"46e5b3d1-37a6-48fa-9d76-170d56582163\" (UID: \"46e5b3d1-37a6-48fa-9d76-170d56582163\") " Jul 28 12:39:46.692108 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:46.692001 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhzfp\" (UniqueName: \"kubernetes.io/projected/46e5b3d1-37a6-48fa-9d76-170d56582163-kube-api-access-jhzfp\") pod \"46e5b3d1-37a6-48fa-9d76-170d56582163\" (UID: \"46e5b3d1-37a6-48fa-9d76-170d56582163\") " Jul 28 12:39:46.692108 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:46.692062 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46e5b3d1-37a6-48fa-9d76-170d56582163-bundle\") pod \"46e5b3d1-37a6-48fa-9d76-170d56582163\" (UID: \"46e5b3d1-37a6-48fa-9d76-170d56582163\") " Jul 28 12:39:46.692777 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:46.692750 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e5b3d1-37a6-48fa-9d76-170d56582163-bundle" (OuterVolumeSpecName: "bundle") pod "46e5b3d1-37a6-48fa-9d76-170d56582163" (UID: "46e5b3d1-37a6-48fa-9d76-170d56582163"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:39:46.694162 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:46.694135 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e5b3d1-37a6-48fa-9d76-170d56582163-kube-api-access-jhzfp" (OuterVolumeSpecName: "kube-api-access-jhzfp") pod "46e5b3d1-37a6-48fa-9d76-170d56582163" (UID: "46e5b3d1-37a6-48fa-9d76-170d56582163"). InnerVolumeSpecName "kube-api-access-jhzfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:39:46.700546 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:46.700524 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e5b3d1-37a6-48fa-9d76-170d56582163-util" (OuterVolumeSpecName: "util") pod "46e5b3d1-37a6-48fa-9d76-170d56582163" (UID: "46e5b3d1-37a6-48fa-9d76-170d56582163"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:39:46.793086 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:46.793062 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46e5b3d1-37a6-48fa-9d76-170d56582163-util\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:39:46.793086 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:46.793083 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhzfp\" (UniqueName: \"kubernetes.io/projected/46e5b3d1-37a6-48fa-9d76-170d56582163-kube-api-access-jhzfp\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:39:46.793213 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:46.793095 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46e5b3d1-37a6-48fa-9d76-170d56582163-bundle\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:39:47.437530 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:47.437490 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" event={"ID":"46e5b3d1-37a6-48fa-9d76-170d56582163","Type":"ContainerDied","Data":"8401984e6ad72f8e5ff98ae222551db45ceb2dea49905e8fbff01a26395296e6"} Jul 28 12:39:47.437530 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:47.437512 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflqf8d" Jul 28 12:39:47.437759 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:47.437542 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8401984e6ad72f8e5ff98ae222551db45ceb2dea49905e8fbff01a26395296e6" Jul 28 12:39:56.650812 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.650783 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd"] Jul 28 12:39:56.651158 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.651058 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46e5b3d1-37a6-48fa-9d76-170d56582163" containerName="pull" Jul 28 12:39:56.651158 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.651069 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="46e5b3d1-37a6-48fa-9d76-170d56582163" containerName="pull" Jul 28 12:39:56.651158 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.651083 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46e5b3d1-37a6-48fa-9d76-170d56582163" containerName="util" Jul 28 12:39:56.651158 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.651089 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="46e5b3d1-37a6-48fa-9d76-170d56582163" containerName="util" Jul 28 12:39:56.651158 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.651097 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46e5b3d1-37a6-48fa-9d76-170d56582163" containerName="extract" Jul 28 12:39:56.651158 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.651103 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="46e5b3d1-37a6-48fa-9d76-170d56582163" containerName="extract" Jul 28 12:39:56.651158 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.651156 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="46e5b3d1-37a6-48fa-9d76-170d56582163" containerName="extract" Jul 28 12:39:56.653300 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.653284 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:56.659972 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.659949 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 28 12:39:56.660059 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.660039 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 28 12:39:56.661482 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.661464 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 28 12:39:56.661684 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.661670 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 28 12:39:56.661735 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.661685 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zw8p2\"" Jul 28 12:39:56.661735 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.661710 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 28 12:39:56.674131 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.674112 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd"] Jul 28 12:39:56.769295 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.769268 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19d50472-8e5a-4a27-b39d-c3a6d632c8b1-cert\") pod \"lws-controller-manager-746578bc67-9lvhd\" (UID: \"19d50472-8e5a-4a27-b39d-c3a6d632c8b1\") " pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:56.769478 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.769301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/19d50472-8e5a-4a27-b39d-c3a6d632c8b1-manager-config\") pod \"lws-controller-manager-746578bc67-9lvhd\" (UID: \"19d50472-8e5a-4a27-b39d-c3a6d632c8b1\") " pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:56.769478 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.769341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/19d50472-8e5a-4a27-b39d-c3a6d632c8b1-metrics-cert\") pod \"lws-controller-manager-746578bc67-9lvhd\" (UID: \"19d50472-8e5a-4a27-b39d-c3a6d632c8b1\") " pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:56.769588 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.769479 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whhhk\" (UniqueName: \"kubernetes.io/projected/19d50472-8e5a-4a27-b39d-c3a6d632c8b1-kube-api-access-whhhk\") pod \"lws-controller-manager-746578bc67-9lvhd\" (UID: \"19d50472-8e5a-4a27-b39d-c3a6d632c8b1\") " pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:56.870633 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.870610 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whhhk\" (UniqueName: \"kubernetes.io/projected/19d50472-8e5a-4a27-b39d-c3a6d632c8b1-kube-api-access-whhhk\") pod \"lws-controller-manager-746578bc67-9lvhd\" (UID: \"19d50472-8e5a-4a27-b39d-c3a6d632c8b1\") " pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:56.870754 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.870641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19d50472-8e5a-4a27-b39d-c3a6d632c8b1-cert\") pod \"lws-controller-manager-746578bc67-9lvhd\" (UID: \"19d50472-8e5a-4a27-b39d-c3a6d632c8b1\") " pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:56.870754 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.870665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/19d50472-8e5a-4a27-b39d-c3a6d632c8b1-manager-config\") pod \"lws-controller-manager-746578bc67-9lvhd\" (UID: \"19d50472-8e5a-4a27-b39d-c3a6d632c8b1\") " pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:56.870754 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.870723 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/19d50472-8e5a-4a27-b39d-c3a6d632c8b1-metrics-cert\") pod \"lws-controller-manager-746578bc67-9lvhd\" (UID: \"19d50472-8e5a-4a27-b39d-c3a6d632c8b1\") " pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:56.871489 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.871388 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/19d50472-8e5a-4a27-b39d-c3a6d632c8b1-manager-config\") pod \"lws-controller-manager-746578bc67-9lvhd\" (UID: \"19d50472-8e5a-4a27-b39d-c3a6d632c8b1\") " pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:56.873223 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.873204 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/19d50472-8e5a-4a27-b39d-c3a6d632c8b1-metrics-cert\") pod \"lws-controller-manager-746578bc67-9lvhd\" (UID: \"19d50472-8e5a-4a27-b39d-c3a6d632c8b1\") " pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:56.873415 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.873395 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19d50472-8e5a-4a27-b39d-c3a6d632c8b1-cert\") pod \"lws-controller-manager-746578bc67-9lvhd\" (UID: \"19d50472-8e5a-4a27-b39d-c3a6d632c8b1\") " pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:56.880803 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.880774 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whhhk\" (UniqueName: \"kubernetes.io/projected/19d50472-8e5a-4a27-b39d-c3a6d632c8b1-kube-api-access-whhhk\") pod \"lws-controller-manager-746578bc67-9lvhd\" (UID: \"19d50472-8e5a-4a27-b39d-c3a6d632c8b1\") " pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:56.962202 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:56.962131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:57.098110 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:57.098089 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd"] Jul 28 12:39:57.099844 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:39:57.099818 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d50472_8e5a_4a27_b39d_c3a6d632c8b1.slice/crio-89d1a2df139751dc83081e3fd1980dd0a7dd4826f98e1afa5f5445c3d56cacda WatchSource:0}: Error finding container 89d1a2df139751dc83081e3fd1980dd0a7dd4826f98e1afa5f5445c3d56cacda: Status 404 returned error can't find the container with id 89d1a2df139751dc83081e3fd1980dd0a7dd4826f98e1afa5f5445c3d56cacda Jul 28 12:39:57.487627 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:57.487595 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" event={"ID":"19d50472-8e5a-4a27-b39d-c3a6d632c8b1","Type":"ContainerStarted","Data":"89d1a2df139751dc83081e3fd1980dd0a7dd4826f98e1afa5f5445c3d56cacda"} Jul 28 12:39:59.402026 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.401995 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs"] Jul 28 12:39:59.404162 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.404147 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.407245 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.407225 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 12:39:59.407245 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.407233 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 12:39:59.407513 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.407499 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fv4tk\"" Jul 28 12:39:59.407572 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.407555 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 12:39:59.410183 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.410166 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 12:39:59.423948 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.423922 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs"] Jul 28 12:39:59.496469 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.496410 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" event={"ID":"19d50472-8e5a-4a27-b39d-c3a6d632c8b1","Type":"ContainerStarted","Data":"b1bb164774ff9466107a5518784b9517871edcf4e471bae0e5740cd41d696fdd"} Jul 28 12:39:59.496627 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.496576 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:39:59.518634 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.518591 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" podStartSLOduration=2.113299645 podStartE2EDuration="3.51857602s" totalImagesPullingTime="1.405276375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:39:59.518066272 +0000 UTC m=+584.777357037" watchObservedRunningTime="2026-07-28 12:39:59.51857602 +0000 UTC m=+584.777866760" Jul 28 12:39:59.594630 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.594602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8q2\" (UniqueName: \"kubernetes.io/projected/0dd05d93-2a5f-41c7-8996-e9560b576309-kube-api-access-zh8q2\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.594800 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.594765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0dd05d93-2a5f-41c7-8996-e9560b576309-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.594881 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.594850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dd05d93-2a5f-41c7-8996-e9560b576309-tmp\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.594942 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.594882 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0dd05d93-2a5f-41c7-8996-e9560b576309-webhook-cert\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.594942 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.594903 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0dd05d93-2a5f-41c7-8996-e9560b576309-manifests\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.696036 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.695976 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dd05d93-2a5f-41c7-8996-e9560b576309-tmp\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.696036 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.696006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0dd05d93-2a5f-41c7-8996-e9560b576309-webhook-cert\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.696036 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.696026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0dd05d93-2a5f-41c7-8996-e9560b576309-manifests\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.696224 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.696042 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8q2\" (UniqueName: \"kubernetes.io/projected/0dd05d93-2a5f-41c7-8996-e9560b576309-kube-api-access-zh8q2\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.696224 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.696086 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0dd05d93-2a5f-41c7-8996-e9560b576309-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.696501 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.696481 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0dd05d93-2a5f-41c7-8996-e9560b576309-manifests\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.698466 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.698433 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dd05d93-2a5f-41c7-8996-e9560b576309-tmp\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.698576 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.698560 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0dd05d93-2a5f-41c7-8996-e9560b576309-webhook-cert\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.698612 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.698585 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0dd05d93-2a5f-41c7-8996-e9560b576309-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.704617 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.704600 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8q2\" (UniqueName: \"kubernetes.io/projected/0dd05d93-2a5f-41c7-8996-e9560b576309-kube-api-access-zh8q2\") pod \"opendatahub-operator-controller-manager-7974886bdb-72sjs\" (UID: \"0dd05d93-2a5f-41c7-8996-e9560b576309\") " pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.714500 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.714475 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:39:59.866817 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:39:59.866777 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs"] Jul 28 12:39:59.874079 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:39:59.874047 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd05d93_2a5f_41c7_8996_e9560b576309.slice/crio-7d3f7f0059c17cc08ea9b81b438e8660c0baf18aaaf6348fcd753c8d39ff9d34 WatchSource:0}: Error finding container 7d3f7f0059c17cc08ea9b81b438e8660c0baf18aaaf6348fcd753c8d39ff9d34: Status 404 returned error can't find the container with id 7d3f7f0059c17cc08ea9b81b438e8660c0baf18aaaf6348fcd753c8d39ff9d34 Jul 28 12:40:00.503693 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:00.503654 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" event={"ID":"0dd05d93-2a5f-41c7-8996-e9560b576309","Type":"ContainerStarted","Data":"7d3f7f0059c17cc08ea9b81b438e8660c0baf18aaaf6348fcd753c8d39ff9d34"} Jul 28 12:40:03.519197 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:03.519157 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" event={"ID":"0dd05d93-2a5f-41c7-8996-e9560b576309","Type":"ContainerStarted","Data":"7efe3308011cf90b1c05e5f8f1a25bbcf002df1ec65594372b29f64137aebf88"} Jul 28 12:40:04.525866 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:04.525784 2573 generic.go:356] "Generic (PLEG): container finished" podID="0dd05d93-2a5f-41c7-8996-e9560b576309" containerID="7efe3308011cf90b1c05e5f8f1a25bbcf002df1ec65594372b29f64137aebf88" exitCode=0 Jul 28 12:40:04.525866 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:04.525836 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" event={"ID":"0dd05d93-2a5f-41c7-8996-e9560b576309","Type":"ContainerDied","Data":"7efe3308011cf90b1c05e5f8f1a25bbcf002df1ec65594372b29f64137aebf88"} Jul 28 12:40:05.533070 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:05.533031 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" event={"ID":"0dd05d93-2a5f-41c7-8996-e9560b576309","Type":"ContainerStarted","Data":"50fd03ad8750c8b4abc1f3830abe315ccd3444aafc5e8e323912b57775be351d"} Jul 28 12:40:05.533599 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:05.533131 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:40:05.556327 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:05.556283 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" podStartSLOduration=1.994177171 podStartE2EDuration="6.556268163s" totalImagesPullingTime="3.488744074s" totalInitContainerRuntime="1.073346918s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:39:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:40:05.55428172 +0000 UTC m=+590.813572497" watchObservedRunningTime="2026-07-28 12:40:05.556268163 +0000 UTC m=+590.815558942" Jul 28 12:40:10.506373 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:10.506341 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-746578bc67-9lvhd" Jul 28 12:40:16.540063 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:16.540029 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7974886bdb-72sjs" Jul 28 12:40:20.128304 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.128264 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57886bb876-gq5dz"] Jul 28 12:40:20.205236 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.205199 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57886bb876-gq5dz"] Jul 28 12:40:20.205407 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.205303 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.260990 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.260959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52c70995-54d6-41b2-8e0c-2b08959bfe15-console-oauth-config\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.261152 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.260996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52c70995-54d6-41b2-8e0c-2b08959bfe15-trusted-ca-bundle\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.261152 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.261017 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52c70995-54d6-41b2-8e0c-2b08959bfe15-oauth-serving-cert\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.261152 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.261033 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52c70995-54d6-41b2-8e0c-2b08959bfe15-console-config\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.261152 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.261140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52c70995-54d6-41b2-8e0c-2b08959bfe15-console-serving-cert\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.261324 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.261179 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52c70995-54d6-41b2-8e0c-2b08959bfe15-service-ca\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.261324 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.261253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl5ds\" (UniqueName: \"kubernetes.io/projected/52c70995-54d6-41b2-8e0c-2b08959bfe15-kube-api-access-xl5ds\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.361949 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.361915 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52c70995-54d6-41b2-8e0c-2b08959bfe15-service-ca\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.362116 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.361966 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl5ds\" (UniqueName: \"kubernetes.io/projected/52c70995-54d6-41b2-8e0c-2b08959bfe15-kube-api-access-xl5ds\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.362116 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.361999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52c70995-54d6-41b2-8e0c-2b08959bfe15-console-oauth-config\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.362116 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.362024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52c70995-54d6-41b2-8e0c-2b08959bfe15-trusted-ca-bundle\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.362116 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.362045 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52c70995-54d6-41b2-8e0c-2b08959bfe15-oauth-serving-cert\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.362116 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.362061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52c70995-54d6-41b2-8e0c-2b08959bfe15-console-config\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.362116 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.362105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52c70995-54d6-41b2-8e0c-2b08959bfe15-console-serving-cert\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.362817 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.362790 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52c70995-54d6-41b2-8e0c-2b08959bfe15-oauth-serving-cert\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.362938 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.362791 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52c70995-54d6-41b2-8e0c-2b08959bfe15-service-ca\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.362938 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.362886 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52c70995-54d6-41b2-8e0c-2b08959bfe15-trusted-ca-bundle\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.363112 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.363089 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52c70995-54d6-41b2-8e0c-2b08959bfe15-console-config\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.364652 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.364631 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52c70995-54d6-41b2-8e0c-2b08959bfe15-console-oauth-config\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.364771 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.364755 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52c70995-54d6-41b2-8e0c-2b08959bfe15-console-serving-cert\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.374161 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.374134 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl5ds\" (UniqueName: \"kubernetes.io/projected/52c70995-54d6-41b2-8e0c-2b08959bfe15-kube-api-access-xl5ds\") pod \"console-57886bb876-gq5dz\" (UID: \"52c70995-54d6-41b2-8e0c-2b08959bfe15\") " pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.515605 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.515572 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:20.672358 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:20.672239 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57886bb876-gq5dz"] Jul 28 12:40:20.675130 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:40:20.675104 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c70995_54d6_41b2_8e0c_2b08959bfe15.slice/crio-5369aa6527d350048b3f060a1529d9c93ea134034f56ea1d3226c0724600c7f7 WatchSource:0}: Error finding container 5369aa6527d350048b3f060a1529d9c93ea134034f56ea1d3226c0724600c7f7: Status 404 returned error can't find the container with id 5369aa6527d350048b3f060a1529d9c93ea134034f56ea1d3226c0724600c7f7 Jul 28 12:40:21.611294 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:21.611250 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57886bb876-gq5dz" event={"ID":"52c70995-54d6-41b2-8e0c-2b08959bfe15","Type":"ContainerStarted","Data":"aee7a37af3fd88e0bd75f8562a152a5170f8196e773c13645d74b9f13d9045e7"} Jul 28 12:40:21.611747 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:21.611316 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57886bb876-gq5dz" event={"ID":"52c70995-54d6-41b2-8e0c-2b08959bfe15","Type":"ContainerStarted","Data":"5369aa6527d350048b3f060a1529d9c93ea134034f56ea1d3226c0724600c7f7"} Jul 28 12:40:21.631460 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:21.631387 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57886bb876-gq5dz" podStartSLOduration=1.63136606 podStartE2EDuration="1.63136606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:40:21.629466293 +0000 UTC m=+606.888762389" watchObservedRunningTime="2026-07-28 12:40:21.63136606 +0000 UTC m=+606.890656803" Jul 28 12:40:26.737119 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.737075 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx"] Jul 28 12:40:26.740287 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.740266 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.744320 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.744298 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 12:40:26.744503 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.744377 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 28 12:40:26.744503 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.744495 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 12:40:26.744659 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.744376 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 28 12:40:26.744659 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.744414 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 28 12:40:26.745088 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.745071 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h6vl8\"" Jul 28 12:40:26.745175 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.745079 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 12:40:26.755944 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.755923 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx"] Jul 28 12:40:26.816583 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.816551 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20ed721-ef7e-47eb-ad07-1f0a28163f89-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.816583 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.816579 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b20ed721-ef7e-47eb-ad07-1f0a28163f89-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.816758 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.816611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20ed721-ef7e-47eb-ad07-1f0a28163f89-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.816758 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.816658 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b20ed721-ef7e-47eb-ad07-1f0a28163f89-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.816758 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.816689 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b20ed721-ef7e-47eb-ad07-1f0a28163f89-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.816758 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.816714 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b20ed721-ef7e-47eb-ad07-1f0a28163f89-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.816758 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.816732 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b65qw\" (UniqueName: \"kubernetes.io/projected/b20ed721-ef7e-47eb-ad07-1f0a28163f89-kube-api-access-b65qw\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.917863 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.917833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20ed721-ef7e-47eb-ad07-1f0a28163f89-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.917863 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.917863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b20ed721-ef7e-47eb-ad07-1f0a28163f89-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.918057 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.917903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20ed721-ef7e-47eb-ad07-1f0a28163f89-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.918057 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.917935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b20ed721-ef7e-47eb-ad07-1f0a28163f89-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.918057 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.917978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b20ed721-ef7e-47eb-ad07-1f0a28163f89-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.918057 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.918003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b20ed721-ef7e-47eb-ad07-1f0a28163f89-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.918057 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.918024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b65qw\" (UniqueName: \"kubernetes.io/projected/b20ed721-ef7e-47eb-ad07-1f0a28163f89-kube-api-access-b65qw\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.918808 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.918696 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b20ed721-ef7e-47eb-ad07-1f0a28163f89-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.920298 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.920271 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b20ed721-ef7e-47eb-ad07-1f0a28163f89-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.920766 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.920741 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20ed721-ef7e-47eb-ad07-1f0a28163f89-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.921026 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.921002 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b20ed721-ef7e-47eb-ad07-1f0a28163f89-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.921067 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.921010 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b20ed721-ef7e-47eb-ad07-1f0a28163f89-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.927970 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.927939 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20ed721-ef7e-47eb-ad07-1f0a28163f89-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:26.928384 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:26.928360 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b65qw\" (UniqueName: \"kubernetes.io/projected/b20ed721-ef7e-47eb-ad07-1f0a28163f89-kube-api-access-b65qw\") pod \"istiod-openshift-gateway-5677ccb49b-jqjlx\" (UID: \"b20ed721-ef7e-47eb-ad07-1f0a28163f89\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:27.051932 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:27.051881 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:27.209350 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:27.209320 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx"] Jul 28 12:40:27.210611 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:40:27.210575 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20ed721_ef7e_47eb_ad07_1f0a28163f89.slice/crio-6eb8a7497886e9c66a96f92c48e946bc1c80d60f9b94d2dd48e410e4ae28394b WatchSource:0}: Error finding container 6eb8a7497886e9c66a96f92c48e946bc1c80d60f9b94d2dd48e410e4ae28394b: Status 404 returned error can't find the container with id 6eb8a7497886e9c66a96f92c48e946bc1c80d60f9b94d2dd48e410e4ae28394b Jul 28 12:40:27.639680 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:27.639645 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" event={"ID":"b20ed721-ef7e-47eb-ad07-1f0a28163f89","Type":"ContainerStarted","Data":"6eb8a7497886e9c66a96f92c48e946bc1c80d60f9b94d2dd48e410e4ae28394b"} Jul 28 12:40:29.642992 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:29.642955 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Jul 28 12:40:29.643348 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:29.643028 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Jul 28 12:40:30.516750 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:30.516720 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:30.516902 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:30.516777 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:30.521221 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:30.521202 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:30.655056 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:30.655027 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" event={"ID":"b20ed721-ef7e-47eb-ad07-1f0a28163f89","Type":"ContainerStarted","Data":"878137e7c14c49b208c2b8a035ef93500025513faaa30ca4ec49710f1290abd5"} Jul 28 12:40:30.658742 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:30.658722 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57886bb876-gq5dz" Jul 28 12:40:30.686659 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:30.686615 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" podStartSLOduration=2.256820143 podStartE2EDuration="4.686600922s" totalImagesPullingTime="2.429780779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:40:30.684994368 +0000 UTC m=+615.944285108" watchObservedRunningTime="2026-07-28 12:40:30.686600922 +0000 UTC m=+615.945891662" Jul 28 12:40:30.794396 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:30.794318 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4dbddb46-n2z8h"] Jul 28 12:40:31.659551 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:31.659512 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:31.661046 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:31.661018 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jqjlx" Jul 28 12:40:55.812771 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:55.812713 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c4dbddb46-n2z8h" podUID="16f63b97-a829-4e6d-861b-3d29cbdcd551" containerName="console" containerID="cri-o://6cdc695cd979ef7918e10793b10062f47b68bbe3d967de20aa62fec1e6d28bb9" gracePeriod=15 Jul 28 12:40:56.053223 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.053200 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:40:56.063200 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063145 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-service-ca\") pod \"16f63b97-a829-4e6d-861b-3d29cbdcd551\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " Jul 28 12:40:56.063200 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063196 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-oauth-serving-cert\") pod \"16f63b97-a829-4e6d-861b-3d29cbdcd551\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " Jul 28 12:40:56.063383 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063214 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gmws\" (UniqueName: \"kubernetes.io/projected/16f63b97-a829-4e6d-861b-3d29cbdcd551-kube-api-access-7gmws\") pod \"16f63b97-a829-4e6d-861b-3d29cbdcd551\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " Jul 28 12:40:56.063383 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063238 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-config\") pod \"16f63b97-a829-4e6d-861b-3d29cbdcd551\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " Jul 28 12:40:56.063383 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063270 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-trusted-ca-bundle\") pod \"16f63b97-a829-4e6d-861b-3d29cbdcd551\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " Jul 28 12:40:56.063383 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063288 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-serving-cert\") pod \"16f63b97-a829-4e6d-861b-3d29cbdcd551\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " Jul 28 12:40:56.063615 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063488 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-oauth-config\") pod \"16f63b97-a829-4e6d-861b-3d29cbdcd551\" (UID: \"16f63b97-a829-4e6d-861b-3d29cbdcd551\") " Jul 28 12:40:56.063615 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063584 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-service-ca" (OuterVolumeSpecName: "service-ca") pod "16f63b97-a829-4e6d-861b-3d29cbdcd551" (UID: "16f63b97-a829-4e6d-861b-3d29cbdcd551"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:40:56.063721 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063696 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-config" (OuterVolumeSpecName: "console-config") pod "16f63b97-a829-4e6d-861b-3d29cbdcd551" (UID: "16f63b97-a829-4e6d-861b-3d29cbdcd551"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:40:56.063779 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063749 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "16f63b97-a829-4e6d-861b-3d29cbdcd551" (UID: "16f63b97-a829-4e6d-861b-3d29cbdcd551"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:40:56.063834 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063771 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "16f63b97-a829-4e6d-861b-3d29cbdcd551" (UID: "16f63b97-a829-4e6d-861b-3d29cbdcd551"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:40:56.063915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063873 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-trusted-ca-bundle\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:40:56.063915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063892 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:40:56.063915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063906 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-oauth-serving-cert\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:40:56.064034 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.063918 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-config\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:40:56.065661 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.065634 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "16f63b97-a829-4e6d-861b-3d29cbdcd551" (UID: "16f63b97-a829-4e6d-861b-3d29cbdcd551"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:40:56.065767 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.065654 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "16f63b97-a829-4e6d-861b-3d29cbdcd551" (UID: "16f63b97-a829-4e6d-861b-3d29cbdcd551"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:40:56.065767 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.065715 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f63b97-a829-4e6d-861b-3d29cbdcd551-kube-api-access-7gmws" (OuterVolumeSpecName: "kube-api-access-7gmws") pod "16f63b97-a829-4e6d-861b-3d29cbdcd551" (UID: "16f63b97-a829-4e6d-861b-3d29cbdcd551"). InnerVolumeSpecName "kube-api-access-7gmws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:40:56.164861 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.164824 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gmws\" (UniqueName: \"kubernetes.io/projected/16f63b97-a829-4e6d-861b-3d29cbdcd551-kube-api-access-7gmws\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:40:56.164861 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.164857 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-serving-cert\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:40:56.164861 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.164868 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16f63b97-a829-4e6d-861b-3d29cbdcd551-console-oauth-config\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:40:56.780332 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.780298 2573 generic.go:356] "Generic (PLEG): container finished" podID="16f63b97-a829-4e6d-861b-3d29cbdcd551" containerID="6cdc695cd979ef7918e10793b10062f47b68bbe3d967de20aa62fec1e6d28bb9" exitCode=0 Jul 28 12:40:56.780524 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.780347 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4dbddb46-n2z8h" Jul 28 12:40:56.780524 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.780383 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4dbddb46-n2z8h" event={"ID":"16f63b97-a829-4e6d-861b-3d29cbdcd551","Type":"ContainerDied","Data":"6cdc695cd979ef7918e10793b10062f47b68bbe3d967de20aa62fec1e6d28bb9"} Jul 28 12:40:56.780524 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.780430 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4dbddb46-n2z8h" event={"ID":"16f63b97-a829-4e6d-861b-3d29cbdcd551","Type":"ContainerDied","Data":"cda0d401943bd5694a212946a54bc663cc36ad93d356d907b30409142e9813b5"} Jul 28 12:40:56.780524 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.780465 2573 scope.go:117] "RemoveContainer" containerID="6cdc695cd979ef7918e10793b10062f47b68bbe3d967de20aa62fec1e6d28bb9" Jul 28 12:40:56.789489 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.789470 2573 scope.go:117] "RemoveContainer" containerID="6cdc695cd979ef7918e10793b10062f47b68bbe3d967de20aa62fec1e6d28bb9" Jul 28 12:40:56.789774 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:40:56.789750 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cdc695cd979ef7918e10793b10062f47b68bbe3d967de20aa62fec1e6d28bb9\": container with ID starting with 6cdc695cd979ef7918e10793b10062f47b68bbe3d967de20aa62fec1e6d28bb9 not found: ID does not exist" containerID="6cdc695cd979ef7918e10793b10062f47b68bbe3d967de20aa62fec1e6d28bb9" Jul 28 12:40:56.789849 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.789780 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cdc695cd979ef7918e10793b10062f47b68bbe3d967de20aa62fec1e6d28bb9"} err="failed to get container status \"6cdc695cd979ef7918e10793b10062f47b68bbe3d967de20aa62fec1e6d28bb9\": rpc error: code = NotFound desc = could not find container \"6cdc695cd979ef7918e10793b10062f47b68bbe3d967de20aa62fec1e6d28bb9\": container with ID starting with 6cdc695cd979ef7918e10793b10062f47b68bbe3d967de20aa62fec1e6d28bb9 not found: ID does not exist" Jul 28 12:40:56.805697 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.805673 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4dbddb46-n2z8h"] Jul 28 12:40:56.807220 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:56.807203 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c4dbddb46-n2z8h"] Jul 28 12:40:57.267374 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:40:57.267339 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16f63b97-a829-4e6d-861b-3d29cbdcd551" path="/var/lib/kubelet/pods/16f63b97-a829-4e6d-861b-3d29cbdcd551/volumes" Jul 28 12:41:36.631729 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.631699 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn"] Jul 28 12:41:36.632181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.632025 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16f63b97-a829-4e6d-861b-3d29cbdcd551" containerName="console" Jul 28 12:41:36.632181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.632036 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="16f63b97-a829-4e6d-861b-3d29cbdcd551" containerName="console" Jul 28 12:41:36.632181 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.632089 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="16f63b97-a829-4e6d-861b-3d29cbdcd551" containerName="console" Jul 28 12:41:36.636490 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.636473 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" Jul 28 12:41:36.639256 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.639236 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 12:41:36.640706 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.640689 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tffwh\"" Jul 28 12:41:36.641900 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.641868 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 12:41:36.643353 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.643330 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn"] Jul 28 12:41:36.798298 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.798240 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ca90fd-3305-453e-a8da-c322bf163a6f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn\" (UID: \"e0ca90fd-3305-453e-a8da-c322bf163a6f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" Jul 28 12:41:36.798298 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.798301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdqf9\" (UniqueName: \"kubernetes.io/projected/e0ca90fd-3305-453e-a8da-c322bf163a6f-kube-api-access-xdqf9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn\" (UID: \"e0ca90fd-3305-453e-a8da-c322bf163a6f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" Jul 28 12:41:36.798580 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.798408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ca90fd-3305-453e-a8da-c322bf163a6f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn\" (UID: \"e0ca90fd-3305-453e-a8da-c322bf163a6f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" Jul 28 12:41:36.899610 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.899568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ca90fd-3305-453e-a8da-c322bf163a6f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn\" (UID: \"e0ca90fd-3305-453e-a8da-c322bf163a6f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" Jul 28 12:41:36.899754 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.899649 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ca90fd-3305-453e-a8da-c322bf163a6f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn\" (UID: \"e0ca90fd-3305-453e-a8da-c322bf163a6f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" Jul 28 12:41:36.899754 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.899685 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdqf9\" (UniqueName: \"kubernetes.io/projected/e0ca90fd-3305-453e-a8da-c322bf163a6f-kube-api-access-xdqf9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn\" (UID: \"e0ca90fd-3305-453e-a8da-c322bf163a6f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" Jul 28 12:41:36.900049 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.900026 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ca90fd-3305-453e-a8da-c322bf163a6f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn\" (UID: \"e0ca90fd-3305-453e-a8da-c322bf163a6f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" Jul 28 12:41:36.900117 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.900042 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ca90fd-3305-453e-a8da-c322bf163a6f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn\" (UID: \"e0ca90fd-3305-453e-a8da-c322bf163a6f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" Jul 28 12:41:36.910216 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.910184 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdqf9\" (UniqueName: \"kubernetes.io/projected/e0ca90fd-3305-453e-a8da-c322bf163a6f-kube-api-access-xdqf9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn\" (UID: \"e0ca90fd-3305-453e-a8da-c322bf163a6f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" Jul 28 12:41:36.953930 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:36.953903 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" Jul 28 12:41:37.032959 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.032931 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs"] Jul 28 12:41:37.036601 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.036580 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" Jul 28 12:41:37.044351 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.044315 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs"] Jul 28 12:41:37.090829 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.090804 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn"] Jul 28 12:41:37.092732 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:41:37.092709 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ca90fd_3305_453e_a8da_c322bf163a6f.slice/crio-f215ec12511d6045d082bdcc3c0e186ea0c3f6374b089a02c3dd3ab2546dccdf WatchSource:0}: Error finding container f215ec12511d6045d082bdcc3c0e186ea0c3f6374b089a02c3dd3ab2546dccdf: Status 404 returned error can't find the container with id f215ec12511d6045d082bdcc3c0e186ea0c3f6374b089a02c3dd3ab2546dccdf Jul 28 12:41:37.202421 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.202390 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chtcf\" (UniqueName: \"kubernetes.io/projected/7e4b1835-079b-4e48-88df-794410cb2261-kube-api-access-chtcf\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs\" (UID: \"7e4b1835-079b-4e48-88df-794410cb2261\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" Jul 28 12:41:37.202547 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.202456 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e4b1835-079b-4e48-88df-794410cb2261-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs\" (UID: \"7e4b1835-079b-4e48-88df-794410cb2261\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" Jul 28 12:41:37.202547 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.202484 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e4b1835-079b-4e48-88df-794410cb2261-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs\" (UID: \"7e4b1835-079b-4e48-88df-794410cb2261\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" Jul 28 12:41:37.303235 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.303195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chtcf\" (UniqueName: \"kubernetes.io/projected/7e4b1835-079b-4e48-88df-794410cb2261-kube-api-access-chtcf\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs\" (UID: \"7e4b1835-079b-4e48-88df-794410cb2261\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" Jul 28 12:41:37.303429 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.303242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e4b1835-079b-4e48-88df-794410cb2261-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs\" (UID: \"7e4b1835-079b-4e48-88df-794410cb2261\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" Jul 28 12:41:37.303429 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.303268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e4b1835-079b-4e48-88df-794410cb2261-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs\" (UID: \"7e4b1835-079b-4e48-88df-794410cb2261\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" Jul 28 12:41:37.303639 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.303616 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e4b1835-079b-4e48-88df-794410cb2261-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs\" (UID: \"7e4b1835-079b-4e48-88df-794410cb2261\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" Jul 28 12:41:37.303710 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.303688 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e4b1835-079b-4e48-88df-794410cb2261-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs\" (UID: \"7e4b1835-079b-4e48-88df-794410cb2261\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" Jul 28 12:41:37.312672 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.312650 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chtcf\" (UniqueName: \"kubernetes.io/projected/7e4b1835-079b-4e48-88df-794410cb2261-kube-api-access-chtcf\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs\" (UID: \"7e4b1835-079b-4e48-88df-794410cb2261\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" Jul 28 12:41:37.350665 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.350637 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" Jul 28 12:41:37.436219 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.436186 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf"] Jul 28 12:41:37.439949 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.439927 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" Jul 28 12:41:37.448270 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.448249 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf"] Jul 28 12:41:37.483497 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.483474 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs"] Jul 28 12:41:37.484993 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:41:37.484971 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4b1835_079b_4e48_88df_794410cb2261.slice/crio-b4e4427fc9805a2abcb07eae6b1e89e74fbaf2d95cebb29fc043b59afa54202f WatchSource:0}: Error finding container b4e4427fc9805a2abcb07eae6b1e89e74fbaf2d95cebb29fc043b59afa54202f: Status 404 returned error can't find the container with id b4e4427fc9805a2abcb07eae6b1e89e74fbaf2d95cebb29fc043b59afa54202f Jul 28 12:41:37.605872 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.605845 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf\" (UID: \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" Jul 28 12:41:37.606003 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.605904 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf\" (UID: \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" Jul 28 12:41:37.606003 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.605957 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tplz7\" (UniqueName: \"kubernetes.io/projected/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-kube-api-access-tplz7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf\" (UID: \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" Jul 28 12:41:37.706645 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.706561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tplz7\" (UniqueName: \"kubernetes.io/projected/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-kube-api-access-tplz7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf\" (UID: \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" Jul 28 12:41:37.706645 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.706612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf\" (UID: \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" Jul 28 12:41:37.707031 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.706656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf\" (UID: \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" Jul 28 12:41:37.707031 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.706983 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf\" (UID: \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" Jul 28 12:41:37.707031 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.707022 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf\" (UID: \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" Jul 28 12:41:37.715324 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.715291 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tplz7\" (UniqueName: \"kubernetes.io/projected/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-kube-api-access-tplz7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf\" (UID: \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" Jul 28 12:41:37.751335 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.751308 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" Jul 28 12:41:37.828584 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.828556 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj"] Jul 28 12:41:37.832163 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.832146 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" Jul 28 12:41:37.840976 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.840545 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj"] Jul 28 12:41:37.888920 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.888898 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf"] Jul 28 12:41:37.890461 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:41:37.890416 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9b484d_2f1a_4429_97b9_a13df4c3d9d3.slice/crio-715a1b9d380e5fc012991dbe1ea5b48add8c3ecddfc1c040a3cca68ecda3d625 WatchSource:0}: Error finding container 715a1b9d380e5fc012991dbe1ea5b48add8c3ecddfc1c040a3cca68ecda3d625: Status 404 returned error can't find the container with id 715a1b9d380e5fc012991dbe1ea5b48add8c3ecddfc1c040a3cca68ecda3d625 Jul 28 12:41:37.969180 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.969100 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" event={"ID":"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3","Type":"ContainerStarted","Data":"1e22ece76ce96ff8a52682b8dda5350d799721cd2966abb75c9dbb226d6ab575"} Jul 28 12:41:37.969180 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.969160 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" event={"ID":"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3","Type":"ContainerStarted","Data":"715a1b9d380e5fc012991dbe1ea5b48add8c3ecddfc1c040a3cca68ecda3d625"} Jul 28 12:41:37.970832 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.970808 2573 generic.go:356] "Generic (PLEG): container finished" podID="7e4b1835-079b-4e48-88df-794410cb2261" containerID="f4e2ae5e859096324ddc47081fb0c620abaca64c90f60278031bd936e70c4b91" exitCode=0 Jul 28 12:41:37.970964 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.970907 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" event={"ID":"7e4b1835-079b-4e48-88df-794410cb2261","Type":"ContainerDied","Data":"f4e2ae5e859096324ddc47081fb0c620abaca64c90f60278031bd936e70c4b91"} Jul 28 12:41:37.970964 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.970945 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" event={"ID":"7e4b1835-079b-4e48-88df-794410cb2261","Type":"ContainerStarted","Data":"b4e4427fc9805a2abcb07eae6b1e89e74fbaf2d95cebb29fc043b59afa54202f"} Jul 28 12:41:37.971306 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.971287 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 12:41:37.972782 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.972754 2573 generic.go:356] "Generic (PLEG): container finished" podID="e0ca90fd-3305-453e-a8da-c322bf163a6f" containerID="a0de10464ed71fc111374d8ed576e795b83f63e5bd3f127173e7783b3237ee69" exitCode=0 Jul 28 12:41:37.972858 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.972769 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" event={"ID":"e0ca90fd-3305-453e-a8da-c322bf163a6f","Type":"ContainerDied","Data":"a0de10464ed71fc111374d8ed576e795b83f63e5bd3f127173e7783b3237ee69"} Jul 28 12:41:37.972858 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:37.972820 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" event={"ID":"e0ca90fd-3305-453e-a8da-c322bf163a6f","Type":"ContainerStarted","Data":"f215ec12511d6045d082bdcc3c0e186ea0c3f6374b089a02c3dd3ab2546dccdf"} Jul 28 12:41:38.009605 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.009582 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dce6f400-0ed3-4fdc-8916-a970b4aeca78-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj\" (UID: \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" Jul 28 12:41:38.009705 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.009621 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dce6f400-0ed3-4fdc-8916-a970b4aeca78-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj\" (UID: \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" Jul 28 12:41:38.009705 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.009648 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlffh\" (UniqueName: \"kubernetes.io/projected/dce6f400-0ed3-4fdc-8916-a970b4aeca78-kube-api-access-jlffh\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj\" (UID: \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" Jul 28 12:41:38.110601 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.110570 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dce6f400-0ed3-4fdc-8916-a970b4aeca78-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj\" (UID: \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" Jul 28 12:41:38.110760 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.110619 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dce6f400-0ed3-4fdc-8916-a970b4aeca78-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj\" (UID: \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" Jul 28 12:41:38.110760 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.110658 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlffh\" (UniqueName: \"kubernetes.io/projected/dce6f400-0ed3-4fdc-8916-a970b4aeca78-kube-api-access-jlffh\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj\" (UID: \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" Jul 28 12:41:38.111018 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.110984 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dce6f400-0ed3-4fdc-8916-a970b4aeca78-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj\" (UID: \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" Jul 28 12:41:38.111087 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.111032 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dce6f400-0ed3-4fdc-8916-a970b4aeca78-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj\" (UID: \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" Jul 28 12:41:38.119795 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.119771 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlffh\" (UniqueName: \"kubernetes.io/projected/dce6f400-0ed3-4fdc-8916-a970b4aeca78-kube-api-access-jlffh\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj\" (UID: \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" Jul 28 12:41:38.144622 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.144595 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" Jul 28 12:41:38.277225 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.277063 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj"] Jul 28 12:41:38.279742 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:41:38.279716 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce6f400_0ed3_4fdc_8916_a970b4aeca78.slice/crio-c06a49aa5309f97643e982c527cedd0fd0142af9864c5fc620a491c9bb1bc42c WatchSource:0}: Error finding container c06a49aa5309f97643e982c527cedd0fd0142af9864c5fc620a491c9bb1bc42c: Status 404 returned error can't find the container with id c06a49aa5309f97643e982c527cedd0fd0142af9864c5fc620a491c9bb1bc42c Jul 28 12:41:38.980216 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.980182 2573 generic.go:356] "Generic (PLEG): container finished" podID="1a9b484d-2f1a-4429-97b9-a13df4c3d9d3" containerID="1e22ece76ce96ff8a52682b8dda5350d799721cd2966abb75c9dbb226d6ab575" exitCode=0 Jul 28 12:41:38.980644 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.980265 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" event={"ID":"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3","Type":"ContainerDied","Data":"1e22ece76ce96ff8a52682b8dda5350d799721cd2966abb75c9dbb226d6ab575"} Jul 28 12:41:38.981989 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.981962 2573 generic.go:356] "Generic (PLEG): container finished" podID="dce6f400-0ed3-4fdc-8916-a970b4aeca78" containerID="91ce555cabcfeae867527dc8054b5662c3057c91ed441caf0e18c413556df4c1" exitCode=0 Jul 28 12:41:38.982130 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.982079 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" event={"ID":"dce6f400-0ed3-4fdc-8916-a970b4aeca78","Type":"ContainerDied","Data":"91ce555cabcfeae867527dc8054b5662c3057c91ed441caf0e18c413556df4c1"} Jul 28 12:41:38.982130 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.982118 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" event={"ID":"dce6f400-0ed3-4fdc-8916-a970b4aeca78","Type":"ContainerStarted","Data":"c06a49aa5309f97643e982c527cedd0fd0142af9864c5fc620a491c9bb1bc42c"} Jul 28 12:41:38.984152 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.984125 2573 generic.go:356] "Generic (PLEG): container finished" podID="e0ca90fd-3305-453e-a8da-c322bf163a6f" containerID="5bf3e570497b7c2c094896039775f405673e16fba8d5cb4428b6c4714b09b865" exitCode=0 Jul 28 12:41:38.984276 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:38.984167 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" event={"ID":"e0ca90fd-3305-453e-a8da-c322bf163a6f","Type":"ContainerDied","Data":"5bf3e570497b7c2c094896039775f405673e16fba8d5cb4428b6c4714b09b865"} Jul 28 12:41:39.991310 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:39.991277 2573 generic.go:356] "Generic (PLEG): container finished" podID="7e4b1835-079b-4e48-88df-794410cb2261" containerID="e9904bd590ba3e4d490022bc693c8acf6442e489669c0b894332f500a32501f3" exitCode=0 Jul 28 12:41:39.991788 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:39.991359 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" event={"ID":"7e4b1835-079b-4e48-88df-794410cb2261","Type":"ContainerDied","Data":"e9904bd590ba3e4d490022bc693c8acf6442e489669c0b894332f500a32501f3"} Jul 28 12:41:39.993479 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:39.993457 2573 generic.go:356] "Generic (PLEG): container finished" podID="e0ca90fd-3305-453e-a8da-c322bf163a6f" containerID="1d047d635b1bc56850ec87dd49162f748ecfc9ac9e8b9589b565054ca3267b88" exitCode=0 Jul 28 12:41:39.993568 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:39.993539 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" event={"ID":"e0ca90fd-3305-453e-a8da-c322bf163a6f","Type":"ContainerDied","Data":"1d047d635b1bc56850ec87dd49162f748ecfc9ac9e8b9589b565054ca3267b88"} Jul 28 12:41:39.995366 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:39.995348 2573 generic.go:356] "Generic (PLEG): container finished" podID="1a9b484d-2f1a-4429-97b9-a13df4c3d9d3" containerID="881cae720b37deb2a4c124d2feb522328e34c459d490c17f4db463657985b970" exitCode=0 Jul 28 12:41:39.995491 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:39.995377 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" event={"ID":"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3","Type":"ContainerDied","Data":"881cae720b37deb2a4c124d2feb522328e34c459d490c17f4db463657985b970"} Jul 28 12:41:41.003885 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.003853 2573 generic.go:356] "Generic (PLEG): container finished" podID="1a9b484d-2f1a-4429-97b9-a13df4c3d9d3" containerID="a7dabdeaba6c34dd347edcc308d23b6c40802e9c2c9975d2321262fbd5f932e5" exitCode=0 Jul 28 12:41:41.004283 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.003923 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" event={"ID":"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3","Type":"ContainerDied","Data":"a7dabdeaba6c34dd347edcc308d23b6c40802e9c2c9975d2321262fbd5f932e5"} Jul 28 12:41:41.005756 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.005729 2573 generic.go:356] "Generic (PLEG): container finished" podID="dce6f400-0ed3-4fdc-8916-a970b4aeca78" containerID="623762ea981ddf4f237fb8b218514fef98b2c8342fe81997766b848e0ae55940" exitCode=0 Jul 28 12:41:41.005867 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.005811 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" event={"ID":"dce6f400-0ed3-4fdc-8916-a970b4aeca78","Type":"ContainerDied","Data":"623762ea981ddf4f237fb8b218514fef98b2c8342fe81997766b848e0ae55940"} Jul 28 12:41:41.007972 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.007949 2573 generic.go:356] "Generic (PLEG): container finished" podID="7e4b1835-079b-4e48-88df-794410cb2261" containerID="c3748f32ff78512b57a13e05d5932a33508694150831e2576468ca39d66645b3" exitCode=0 Jul 28 12:41:41.008089 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.008044 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" event={"ID":"7e4b1835-079b-4e48-88df-794410cb2261","Type":"ContainerDied","Data":"c3748f32ff78512b57a13e05d5932a33508694150831e2576468ca39d66645b3"} Jul 28 12:41:41.149978 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.149955 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" Jul 28 12:41:41.340557 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.340457 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ca90fd-3305-453e-a8da-c322bf163a6f-bundle\") pod \"e0ca90fd-3305-453e-a8da-c322bf163a6f\" (UID: \"e0ca90fd-3305-453e-a8da-c322bf163a6f\") " Jul 28 12:41:41.340715 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.340562 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ca90fd-3305-453e-a8da-c322bf163a6f-util\") pod \"e0ca90fd-3305-453e-a8da-c322bf163a6f\" (UID: \"e0ca90fd-3305-453e-a8da-c322bf163a6f\") " Jul 28 12:41:41.340715 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.340584 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdqf9\" (UniqueName: \"kubernetes.io/projected/e0ca90fd-3305-453e-a8da-c322bf163a6f-kube-api-access-xdqf9\") pod \"e0ca90fd-3305-453e-a8da-c322bf163a6f\" (UID: \"e0ca90fd-3305-453e-a8da-c322bf163a6f\") " Jul 28 12:41:41.341098 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.341072 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ca90fd-3305-453e-a8da-c322bf163a6f-bundle" (OuterVolumeSpecName: "bundle") pod "e0ca90fd-3305-453e-a8da-c322bf163a6f" (UID: "e0ca90fd-3305-453e-a8da-c322bf163a6f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:41:41.342974 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.342951 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ca90fd-3305-453e-a8da-c322bf163a6f-kube-api-access-xdqf9" (OuterVolumeSpecName: "kube-api-access-xdqf9") pod "e0ca90fd-3305-453e-a8da-c322bf163a6f" (UID: "e0ca90fd-3305-453e-a8da-c322bf163a6f"). InnerVolumeSpecName "kube-api-access-xdqf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:41:41.345968 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.345928 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ca90fd-3305-453e-a8da-c322bf163a6f-util" (OuterVolumeSpecName: "util") pod "e0ca90fd-3305-453e-a8da-c322bf163a6f" (UID: "e0ca90fd-3305-453e-a8da-c322bf163a6f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:41:41.441595 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.441562 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdqf9\" (UniqueName: \"kubernetes.io/projected/e0ca90fd-3305-453e-a8da-c322bf163a6f-kube-api-access-xdqf9\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:41:41.441595 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.441587 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ca90fd-3305-453e-a8da-c322bf163a6f-bundle\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:41:41.441788 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:41.441600 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ca90fd-3305-453e-a8da-c322bf163a6f-util\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:41:42.020328 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.020298 2573 generic.go:356] "Generic (PLEG): container finished" podID="dce6f400-0ed3-4fdc-8916-a970b4aeca78" containerID="51bc26de5029e661a5b3f761212d9cfa9dd7d63e14a37243afddb7bb58c9f286" exitCode=0 Jul 28 12:41:42.020821 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.020380 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" event={"ID":"dce6f400-0ed3-4fdc-8916-a970b4aeca78","Type":"ContainerDied","Data":"51bc26de5029e661a5b3f761212d9cfa9dd7d63e14a37243afddb7bb58c9f286"} Jul 28 12:41:42.022291 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.022273 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" Jul 28 12:41:42.022398 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.022299 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn" event={"ID":"e0ca90fd-3305-453e-a8da-c322bf163a6f","Type":"ContainerDied","Data":"f215ec12511d6045d082bdcc3c0e186ea0c3f6374b089a02c3dd3ab2546dccdf"} Jul 28 12:41:42.022398 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.022335 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f215ec12511d6045d082bdcc3c0e186ea0c3f6374b089a02c3dd3ab2546dccdf" Jul 28 12:41:42.163532 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.163510 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" Jul 28 12:41:42.189783 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.189761 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" Jul 28 12:41:42.349082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.348988 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-util\") pod \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\" (UID: \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\") " Jul 28 12:41:42.349082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.349037 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chtcf\" (UniqueName: \"kubernetes.io/projected/7e4b1835-079b-4e48-88df-794410cb2261-kube-api-access-chtcf\") pod \"7e4b1835-079b-4e48-88df-794410cb2261\" (UID: \"7e4b1835-079b-4e48-88df-794410cb2261\") " Jul 28 12:41:42.349082 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.349079 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e4b1835-079b-4e48-88df-794410cb2261-util\") pod \"7e4b1835-079b-4e48-88df-794410cb2261\" (UID: \"7e4b1835-079b-4e48-88df-794410cb2261\") " Jul 28 12:41:42.349374 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.349103 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tplz7\" (UniqueName: \"kubernetes.io/projected/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-kube-api-access-tplz7\") pod \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\" (UID: \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\") " Jul 28 12:41:42.349374 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.349133 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e4b1835-079b-4e48-88df-794410cb2261-bundle\") pod \"7e4b1835-079b-4e48-88df-794410cb2261\" (UID: \"7e4b1835-079b-4e48-88df-794410cb2261\") " Jul 28 12:41:42.349374 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.349180 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-bundle\") pod \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\" (UID: \"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3\") " Jul 28 12:41:42.349809 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.349754 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-bundle" (OuterVolumeSpecName: "bundle") pod "1a9b484d-2f1a-4429-97b9-a13df4c3d9d3" (UID: "1a9b484d-2f1a-4429-97b9-a13df4c3d9d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:41:42.349809 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.349753 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4b1835-079b-4e48-88df-794410cb2261-bundle" (OuterVolumeSpecName: "bundle") pod "7e4b1835-079b-4e48-88df-794410cb2261" (UID: "7e4b1835-079b-4e48-88df-794410cb2261"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:41:42.351905 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.351883 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-kube-api-access-tplz7" (OuterVolumeSpecName: "kube-api-access-tplz7") pod "1a9b484d-2f1a-4429-97b9-a13df4c3d9d3" (UID: "1a9b484d-2f1a-4429-97b9-a13df4c3d9d3"). InnerVolumeSpecName "kube-api-access-tplz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:41:42.352005 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.351941 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4b1835-079b-4e48-88df-794410cb2261-kube-api-access-chtcf" (OuterVolumeSpecName: "kube-api-access-chtcf") pod "7e4b1835-079b-4e48-88df-794410cb2261" (UID: "7e4b1835-079b-4e48-88df-794410cb2261"). InnerVolumeSpecName "kube-api-access-chtcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:41:42.354185 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.354163 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-util" (OuterVolumeSpecName: "util") pod "1a9b484d-2f1a-4429-97b9-a13df4c3d9d3" (UID: "1a9b484d-2f1a-4429-97b9-a13df4c3d9d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:41:42.354922 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.354903 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4b1835-079b-4e48-88df-794410cb2261-util" (OuterVolumeSpecName: "util") pod "7e4b1835-079b-4e48-88df-794410cb2261" (UID: "7e4b1835-079b-4e48-88df-794410cb2261"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:41:42.450523 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.450481 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-util\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:41:42.450523 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.450515 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chtcf\" (UniqueName: \"kubernetes.io/projected/7e4b1835-079b-4e48-88df-794410cb2261-kube-api-access-chtcf\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:41:42.450523 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.450529 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e4b1835-079b-4e48-88df-794410cb2261-util\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:41:42.450758 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.450541 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tplz7\" (UniqueName: \"kubernetes.io/projected/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-kube-api-access-tplz7\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:41:42.450758 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.450552 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e4b1835-079b-4e48-88df-794410cb2261-bundle\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:41:42.450758 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:42.450564 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a9b484d-2f1a-4429-97b9-a13df4c3d9d3-bundle\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:41:43.029301 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.029272 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" Jul 28 12:41:43.029751 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.029296 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf" event={"ID":"1a9b484d-2f1a-4429-97b9-a13df4c3d9d3","Type":"ContainerDied","Data":"715a1b9d380e5fc012991dbe1ea5b48add8c3ecddfc1c040a3cca68ecda3d625"} Jul 28 12:41:43.029751 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.029345 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="715a1b9d380e5fc012991dbe1ea5b48add8c3ecddfc1c040a3cca68ecda3d625" Jul 28 12:41:43.031187 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.031161 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" Jul 28 12:41:43.031308 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.031193 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs" event={"ID":"7e4b1835-079b-4e48-88df-794410cb2261","Type":"ContainerDied","Data":"b4e4427fc9805a2abcb07eae6b1e89e74fbaf2d95cebb29fc043b59afa54202f"} Jul 28 12:41:43.031308 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.031233 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4e4427fc9805a2abcb07eae6b1e89e74fbaf2d95cebb29fc043b59afa54202f" Jul 28 12:41:43.164544 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.164519 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" Jul 28 12:41:43.358090 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.358007 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dce6f400-0ed3-4fdc-8916-a970b4aeca78-bundle\") pod \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\" (UID: \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\") " Jul 28 12:41:43.358090 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.358056 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dce6f400-0ed3-4fdc-8916-a970b4aeca78-util\") pod \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\" (UID: \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\") " Jul 28 12:41:43.358315 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.358127 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlffh\" (UniqueName: \"kubernetes.io/projected/dce6f400-0ed3-4fdc-8916-a970b4aeca78-kube-api-access-jlffh\") pod \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\" (UID: \"dce6f400-0ed3-4fdc-8916-a970b4aeca78\") " Jul 28 12:41:43.358558 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.358534 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce6f400-0ed3-4fdc-8916-a970b4aeca78-bundle" (OuterVolumeSpecName: "bundle") pod "dce6f400-0ed3-4fdc-8916-a970b4aeca78" (UID: "dce6f400-0ed3-4fdc-8916-a970b4aeca78"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:41:43.360327 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.360303 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce6f400-0ed3-4fdc-8916-a970b4aeca78-kube-api-access-jlffh" (OuterVolumeSpecName: "kube-api-access-jlffh") pod "dce6f400-0ed3-4fdc-8916-a970b4aeca78" (UID: "dce6f400-0ed3-4fdc-8916-a970b4aeca78"). InnerVolumeSpecName "kube-api-access-jlffh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:41:43.363396 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.363373 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce6f400-0ed3-4fdc-8916-a970b4aeca78-util" (OuterVolumeSpecName: "util") pod "dce6f400-0ed3-4fdc-8916-a970b4aeca78" (UID: "dce6f400-0ed3-4fdc-8916-a970b4aeca78"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:41:43.459318 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.459282 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlffh\" (UniqueName: \"kubernetes.io/projected/dce6f400-0ed3-4fdc-8916-a970b4aeca78-kube-api-access-jlffh\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:41:43.459318 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.459311 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dce6f400-0ed3-4fdc-8916-a970b4aeca78-bundle\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:41:43.459318 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:43.459320 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dce6f400-0ed3-4fdc-8916-a970b4aeca78-util\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:41:44.038516 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:44.038479 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" event={"ID":"dce6f400-0ed3-4fdc-8916-a970b4aeca78","Type":"ContainerDied","Data":"c06a49aa5309f97643e982c527cedd0fd0142af9864c5fc620a491c9bb1bc42c"} Jul 28 12:41:44.038516 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:44.038504 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj" Jul 28 12:41:44.038932 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:41:44.038532 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c06a49aa5309f97643e982c527cedd0fd0142af9864c5fc620a491c9bb1bc42c" Jul 28 12:42:22.382487 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382390 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr"] Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382714 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a9b484d-2f1a-4429-97b9-a13df4c3d9d3" containerName="util" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382726 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a9b484d-2f1a-4429-97b9-a13df4c3d9d3" containerName="util" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382733 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a9b484d-2f1a-4429-97b9-a13df4c3d9d3" containerName="pull" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382739 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a9b484d-2f1a-4429-97b9-a13df4c3d9d3" containerName="pull" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382748 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ca90fd-3305-453e-a8da-c322bf163a6f" containerName="pull" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382753 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ca90fd-3305-453e-a8da-c322bf163a6f" containerName="pull" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382759 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce6f400-0ed3-4fdc-8916-a970b4aeca78" containerName="pull" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382763 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce6f400-0ed3-4fdc-8916-a970b4aeca78" containerName="pull" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382771 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e4b1835-079b-4e48-88df-794410cb2261" containerName="extract" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382776 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e4b1835-079b-4e48-88df-794410cb2261" containerName="extract" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382784 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce6f400-0ed3-4fdc-8916-a970b4aeca78" containerName="util" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382790 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce6f400-0ed3-4fdc-8916-a970b4aeca78" containerName="util" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382796 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ca90fd-3305-453e-a8da-c322bf163a6f" containerName="extract" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382801 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ca90fd-3305-453e-a8da-c322bf163a6f" containerName="extract" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382808 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a9b484d-2f1a-4429-97b9-a13df4c3d9d3" containerName="extract" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382813 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a9b484d-2f1a-4429-97b9-a13df4c3d9d3" containerName="extract" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382822 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e4b1835-079b-4e48-88df-794410cb2261" containerName="pull" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382827 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e4b1835-079b-4e48-88df-794410cb2261" containerName="pull" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382834 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e4b1835-079b-4e48-88df-794410cb2261" containerName="util" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382838 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e4b1835-079b-4e48-88df-794410cb2261" containerName="util" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382844 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce6f400-0ed3-4fdc-8916-a970b4aeca78" containerName="extract" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382850 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce6f400-0ed3-4fdc-8916-a970b4aeca78" containerName="extract" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382856 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ca90fd-3305-453e-a8da-c322bf163a6f" containerName="util" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382862 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ca90fd-3305-453e-a8da-c322bf163a6f" containerName="util" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382912 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce6f400-0ed3-4fdc-8916-a970b4aeca78" containerName="extract" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382918 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0ca90fd-3305-453e-a8da-c322bf163a6f" containerName="extract" Jul 28 12:42:22.382915 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382925 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e4b1835-079b-4e48-88df-794410cb2261" containerName="extract" Jul 28 12:42:22.383713 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.382931 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a9b484d-2f1a-4429-97b9-a13df4c3d9d3" containerName="extract" Jul 28 12:42:22.385892 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.385877 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:22.388814 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.388793 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5k6kz\"" Jul 28 12:42:22.388930 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.388827 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 12:42:22.388930 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.388836 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 12:42:22.398377 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.398356 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr"] Jul 28 12:42:22.469587 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.469553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9458a25b-de42-44d1-96a1-57a640318fcc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-prmdr\" (UID: \"9458a25b-de42-44d1-96a1-57a640318fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:22.469781 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.469594 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8sc6\" (UniqueName: \"kubernetes.io/projected/9458a25b-de42-44d1-96a1-57a640318fcc-kube-api-access-q8sc6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-prmdr\" (UID: \"9458a25b-de42-44d1-96a1-57a640318fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:22.570595 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.570561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9458a25b-de42-44d1-96a1-57a640318fcc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-prmdr\" (UID: \"9458a25b-de42-44d1-96a1-57a640318fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:22.570595 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.570603 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8sc6\" (UniqueName: \"kubernetes.io/projected/9458a25b-de42-44d1-96a1-57a640318fcc-kube-api-access-q8sc6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-prmdr\" (UID: \"9458a25b-de42-44d1-96a1-57a640318fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:22.570994 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.570970 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9458a25b-de42-44d1-96a1-57a640318fcc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-prmdr\" (UID: \"9458a25b-de42-44d1-96a1-57a640318fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:22.579093 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.579066 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8sc6\" (UniqueName: \"kubernetes.io/projected/9458a25b-de42-44d1-96a1-57a640318fcc-kube-api-access-q8sc6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-prmdr\" (UID: \"9458a25b-de42-44d1-96a1-57a640318fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:22.695929 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.695905 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:22.829578 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:22.829554 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr"] Jul 28 12:42:22.831276 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:42:22.831253 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9458a25b_de42_44d1_96a1_57a640318fcc.slice/crio-099571782a08cd81672372851b8a461cdf26b4543b0e24575a4b93d3927aa2aa WatchSource:0}: Error finding container 099571782a08cd81672372851b8a461cdf26b4543b0e24575a4b93d3927aa2aa: Status 404 returned error can't find the container with id 099571782a08cd81672372851b8a461cdf26b4543b0e24575a4b93d3927aa2aa Jul 28 12:42:23.106165 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:23.106076 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr"] Jul 28 12:42:23.112425 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:23.112397 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr"] Jul 28 12:42:23.137652 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:23.137623 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh"] Jul 28 12:42:23.142046 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:23.142030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" Jul 28 12:42:23.163124 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:23.163103 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh"] Jul 28 12:42:23.275352 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:23.275323 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd5gf\" (UniqueName: \"kubernetes.io/projected/6a0a29fb-78e1-45db-92d8-e59a9c62b9fe-kube-api-access-sd5gf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vc6zh\" (UID: \"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" Jul 28 12:42:23.275352 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:23.275358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a0a29fb-78e1-45db-92d8-e59a9c62b9fe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vc6zh\" (UID: \"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" Jul 28 12:42:23.376483 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:23.376400 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd5gf\" (UniqueName: \"kubernetes.io/projected/6a0a29fb-78e1-45db-92d8-e59a9c62b9fe-kube-api-access-sd5gf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vc6zh\" (UID: \"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" Jul 28 12:42:23.376483 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:23.376452 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a0a29fb-78e1-45db-92d8-e59a9c62b9fe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vc6zh\" (UID: \"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" Jul 28 12:42:23.376859 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:23.376836 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a0a29fb-78e1-45db-92d8-e59a9c62b9fe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vc6zh\" (UID: \"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" Jul 28 12:42:23.385119 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:23.385093 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd5gf\" (UniqueName: \"kubernetes.io/projected/6a0a29fb-78e1-45db-92d8-e59a9c62b9fe-kube-api-access-sd5gf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vc6zh\" (UID: \"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" Jul 28 12:42:23.453223 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:23.453194 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" Jul 28 12:42:23.828463 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:23.828424 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh"] Jul 28 12:42:23.906065 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:42:23.906030 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0a29fb_78e1_45db_92d8_e59a9c62b9fe.slice/crio-f6714fb0026149335c94b71c6388de34919e8618de6e555438f18a71601369f3 WatchSource:0}: Error finding container f6714fb0026149335c94b71c6388de34919e8618de6e555438f18a71601369f3: Status 404 returned error can't find the container with id f6714fb0026149335c94b71c6388de34919e8618de6e555438f18a71601369f3 Jul 28 12:42:24.248056 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:24.248015 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" event={"ID":"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe","Type":"ContainerStarted","Data":"f6714fb0026149335c94b71c6388de34919e8618de6e555438f18a71601369f3"} Jul 28 12:42:28.274757 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:28.274689 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" podUID="9458a25b-de42-44d1-96a1-57a640318fcc" containerName="manager" containerID="cri-o://197449d65efdeb376e16fe8d57eb82968793c085fda769b1e3916d7687bd7040" gracePeriod=2 Jul 28 12:42:28.276475 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:28.276421 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" event={"ID":"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe","Type":"ContainerStarted","Data":"c6603dce37ae4d17ddeb271df83c2955b0a8acf4f796b295ef07f9add5d13a6b"} Jul 28 12:42:28.276592 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:28.276574 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" Jul 28 12:42:28.277205 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:42:28.277183 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-prmdr\" is forbidden: User \"system:node:ip-10-0-136-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-145.ec2.internal' and this object" podUID="9458a25b-de42-44d1-96a1-57a640318fcc" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:28.279589 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:42:28.279567 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-prmdr\" is forbidden: User \"system:node:ip-10-0-136-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-145.ec2.internal' and this object" podUID="9458a25b-de42-44d1-96a1-57a640318fcc" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:28.301404 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:28.301346 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" podStartSLOduration=1.926136606 podStartE2EDuration="5.301331015s" totalImagesPullingTime="3.375194409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:42:28.299195975 +0000 UTC m=+733.558486715" watchObservedRunningTime="2026-07-28 12:42:28.301331015 +0000 UTC m=+733.560621757" Jul 28 12:42:28.513603 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:28.513579 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:28.516054 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:42:28.516028 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-prmdr\" is forbidden: User \"system:node:ip-10-0-136-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-145.ec2.internal' and this object" podUID="9458a25b-de42-44d1-96a1-57a640318fcc" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:28.628903 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:28.628814 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9458a25b-de42-44d1-96a1-57a640318fcc-extensions-socket-volume\") pod \"9458a25b-de42-44d1-96a1-57a640318fcc\" (UID: \"9458a25b-de42-44d1-96a1-57a640318fcc\") " Jul 28 12:42:28.629069 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:28.628918 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8sc6\" (UniqueName: \"kubernetes.io/projected/9458a25b-de42-44d1-96a1-57a640318fcc-kube-api-access-q8sc6\") pod \"9458a25b-de42-44d1-96a1-57a640318fcc\" (UID: \"9458a25b-de42-44d1-96a1-57a640318fcc\") " Jul 28 12:42:28.629172 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:28.629142 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9458a25b-de42-44d1-96a1-57a640318fcc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9458a25b-de42-44d1-96a1-57a640318fcc" (UID: "9458a25b-de42-44d1-96a1-57a640318fcc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:42:28.631097 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:28.631073 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9458a25b-de42-44d1-96a1-57a640318fcc-kube-api-access-q8sc6" (OuterVolumeSpecName: "kube-api-access-q8sc6") pod "9458a25b-de42-44d1-96a1-57a640318fcc" (UID: "9458a25b-de42-44d1-96a1-57a640318fcc"). InnerVolumeSpecName "kube-api-access-q8sc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:42:28.730419 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:28.730381 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9458a25b-de42-44d1-96a1-57a640318fcc-extensions-socket-volume\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:42:28.730419 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:28.730410 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8sc6\" (UniqueName: \"kubernetes.io/projected/9458a25b-de42-44d1-96a1-57a640318fcc-kube-api-access-q8sc6\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:42:29.268756 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:29.268723 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9458a25b-de42-44d1-96a1-57a640318fcc" path="/var/lib/kubelet/pods/9458a25b-de42-44d1-96a1-57a640318fcc/volumes" Jul 28 12:42:29.281928 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:29.281900 2573 generic.go:356] "Generic (PLEG): container finished" podID="9458a25b-de42-44d1-96a1-57a640318fcc" containerID="197449d65efdeb376e16fe8d57eb82968793c085fda769b1e3916d7687bd7040" exitCode=2 Jul 28 12:42:29.282227 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:29.281943 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:29.282227 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:29.281991 2573 scope.go:117] "RemoveContainer" containerID="197449d65efdeb376e16fe8d57eb82968793c085fda769b1e3916d7687bd7040" Jul 28 12:42:29.285060 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:42:29.285025 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-prmdr\" is forbidden: User \"system:node:ip-10-0-136-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-145.ec2.internal' and this object" podUID="9458a25b-de42-44d1-96a1-57a640318fcc" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:29.287400 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:42:29.287378 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-prmdr\" is forbidden: User \"system:node:ip-10-0-136-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-145.ec2.internal' and this object" podUID="9458a25b-de42-44d1-96a1-57a640318fcc" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-prmdr" Jul 28 12:42:29.290768 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:29.290756 2573 scope.go:117] "RemoveContainer" containerID="197449d65efdeb376e16fe8d57eb82968793c085fda769b1e3916d7687bd7040" Jul 28 12:42:29.291003 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:42:29.290987 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"197449d65efdeb376e16fe8d57eb82968793c085fda769b1e3916d7687bd7040\": container with ID starting with 197449d65efdeb376e16fe8d57eb82968793c085fda769b1e3916d7687bd7040 not found: ID does not exist" containerID="197449d65efdeb376e16fe8d57eb82968793c085fda769b1e3916d7687bd7040" Jul 28 12:42:29.291048 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:29.291011 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"197449d65efdeb376e16fe8d57eb82968793c085fda769b1e3916d7687bd7040"} err="failed to get container status \"197449d65efdeb376e16fe8d57eb82968793c085fda769b1e3916d7687bd7040\": rpc error: code = NotFound desc = could not find container \"197449d65efdeb376e16fe8d57eb82968793c085fda769b1e3916d7687bd7040\": container with ID starting with 197449d65efdeb376e16fe8d57eb82968793c085fda769b1e3916d7687bd7040 not found: ID does not exist" Jul 28 12:42:39.284291 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:39.284264 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" Jul 28 12:42:56.946640 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:56.946605 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh"] Jul 28 12:42:56.947085 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:56.946864 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" podUID="6a0a29fb-78e1-45db-92d8-e59a9c62b9fe" containerName="manager" containerID="cri-o://c6603dce37ae4d17ddeb271df83c2955b0a8acf4f796b295ef07f9add5d13a6b" gracePeriod=10 Jul 28 12:42:57.194062 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.194041 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" Jul 28 12:42:57.260258 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.260197 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a0a29fb-78e1-45db-92d8-e59a9c62b9fe-extensions-socket-volume\") pod \"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe\" (UID: \"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe\") " Jul 28 12:42:57.260258 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.260239 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd5gf\" (UniqueName: \"kubernetes.io/projected/6a0a29fb-78e1-45db-92d8-e59a9c62b9fe-kube-api-access-sd5gf\") pod \"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe\" (UID: \"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe\") " Jul 28 12:42:57.260792 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.260761 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0a29fb-78e1-45db-92d8-e59a9c62b9fe-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6a0a29fb-78e1-45db-92d8-e59a9c62b9fe" (UID: "6a0a29fb-78e1-45db-92d8-e59a9c62b9fe"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:42:57.262456 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.262415 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0a29fb-78e1-45db-92d8-e59a9c62b9fe-kube-api-access-sd5gf" (OuterVolumeSpecName: "kube-api-access-sd5gf") pod "6a0a29fb-78e1-45db-92d8-e59a9c62b9fe" (UID: "6a0a29fb-78e1-45db-92d8-e59a9c62b9fe"). InnerVolumeSpecName "kube-api-access-sd5gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:42:57.361241 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.361214 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a0a29fb-78e1-45db-92d8-e59a9c62b9fe-extensions-socket-volume\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:42:57.361345 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.361243 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd5gf\" (UniqueName: \"kubernetes.io/projected/6a0a29fb-78e1-45db-92d8-e59a9c62b9fe-kube-api-access-sd5gf\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 12:42:57.417584 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.417556 2573 generic.go:356] "Generic (PLEG): container finished" podID="6a0a29fb-78e1-45db-92d8-e59a9c62b9fe" containerID="c6603dce37ae4d17ddeb271df83c2955b0a8acf4f796b295ef07f9add5d13a6b" exitCode=0 Jul 28 12:42:57.417671 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.417605 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" Jul 28 12:42:57.417671 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.417651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" event={"ID":"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe","Type":"ContainerDied","Data":"c6603dce37ae4d17ddeb271df83c2955b0a8acf4f796b295ef07f9add5d13a6b"} Jul 28 12:42:57.417745 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.417702 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh" event={"ID":"6a0a29fb-78e1-45db-92d8-e59a9c62b9fe","Type":"ContainerDied","Data":"f6714fb0026149335c94b71c6388de34919e8618de6e555438f18a71601369f3"} Jul 28 12:42:57.417745 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.417732 2573 scope.go:117] "RemoveContainer" containerID="c6603dce37ae4d17ddeb271df83c2955b0a8acf4f796b295ef07f9add5d13a6b" Jul 28 12:42:57.426571 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.426549 2573 scope.go:117] "RemoveContainer" containerID="c6603dce37ae4d17ddeb271df83c2955b0a8acf4f796b295ef07f9add5d13a6b" Jul 28 12:42:57.426827 ip-10-0-136-145 kubenswrapper[2573]: E0728 12:42:57.426804 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6603dce37ae4d17ddeb271df83c2955b0a8acf4f796b295ef07f9add5d13a6b\": container with ID starting with c6603dce37ae4d17ddeb271df83c2955b0a8acf4f796b295ef07f9add5d13a6b not found: ID does not exist" containerID="c6603dce37ae4d17ddeb271df83c2955b0a8acf4f796b295ef07f9add5d13a6b" Jul 28 12:42:57.426886 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.426836 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6603dce37ae4d17ddeb271df83c2955b0a8acf4f796b295ef07f9add5d13a6b"} err="failed to get container status \"c6603dce37ae4d17ddeb271df83c2955b0a8acf4f796b295ef07f9add5d13a6b\": rpc error: code = NotFound desc = could not find container \"c6603dce37ae4d17ddeb271df83c2955b0a8acf4f796b295ef07f9add5d13a6b\": container with ID starting with c6603dce37ae4d17ddeb271df83c2955b0a8acf4f796b295ef07f9add5d13a6b not found: ID does not exist" Jul 28 12:42:57.436560 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.436539 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh"] Jul 28 12:42:57.440281 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:57.440260 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vc6zh"] Jul 28 12:42:59.267856 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:42:59.267822 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a0a29fb-78e1-45db-92d8-e59a9c62b9fe" path="/var/lib/kubelet/pods/6a0a29fb-78e1-45db-92d8-e59a9c62b9fe/volumes" Jul 28 12:43:14.489021 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.488985 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:43:14.489405 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.489294 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0a29fb-78e1-45db-92d8-e59a9c62b9fe" containerName="manager" Jul 28 12:43:14.489405 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.489309 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0a29fb-78e1-45db-92d8-e59a9c62b9fe" containerName="manager" Jul 28 12:43:14.489405 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.489329 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9458a25b-de42-44d1-96a1-57a640318fcc" containerName="manager" Jul 28 12:43:14.489405 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.489334 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9458a25b-de42-44d1-96a1-57a640318fcc" containerName="manager" Jul 28 12:43:14.489405 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.489398 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9458a25b-de42-44d1-96a1-57a640318fcc" containerName="manager" Jul 28 12:43:14.489405 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.489406 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0a29fb-78e1-45db-92d8-e59a9c62b9fe" containerName="manager" Jul 28 12:43:14.492549 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.492531 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-5b8ps" Jul 28 12:43:14.495382 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.495364 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 12:43:14.495495 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.495428 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 28 12:43:14.496849 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.496830 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 12:43:14.496952 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.496835 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tffwh\"" Jul 28 12:43:14.509020 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.508997 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:43:14.530180 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.530155 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:43:14.591090 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.591064 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e9e35209-fe83-4acb-9140-2e70b8f129b3-config-file\") pod \"limitador-limitador-54d6d988d5-5b8ps\" (UID: \"e9e35209-fe83-4acb-9140-2e70b8f129b3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5b8ps" Jul 28 12:43:14.591193 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.591099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrfh\" (UniqueName: \"kubernetes.io/projected/e9e35209-fe83-4acb-9140-2e70b8f129b3-kube-api-access-9rrfh\") pod \"limitador-limitador-54d6d988d5-5b8ps\" (UID: \"e9e35209-fe83-4acb-9140-2e70b8f129b3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5b8ps" Jul 28 12:43:14.691826 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.691802 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e9e35209-fe83-4acb-9140-2e70b8f129b3-config-file\") pod \"limitador-limitador-54d6d988d5-5b8ps\" (UID: \"e9e35209-fe83-4acb-9140-2e70b8f129b3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5b8ps" Jul 28 12:43:14.691945 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.691834 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrfh\" (UniqueName: \"kubernetes.io/projected/e9e35209-fe83-4acb-9140-2e70b8f129b3-kube-api-access-9rrfh\") pod \"limitador-limitador-54d6d988d5-5b8ps\" (UID: \"e9e35209-fe83-4acb-9140-2e70b8f129b3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5b8ps" Jul 28 12:43:14.692426 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.692403 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e9e35209-fe83-4acb-9140-2e70b8f129b3-config-file\") pod \"limitador-limitador-54d6d988d5-5b8ps\" (UID: \"e9e35209-fe83-4acb-9140-2e70b8f129b3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5b8ps" Jul 28 12:43:14.700652 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.700631 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrfh\" (UniqueName: \"kubernetes.io/projected/e9e35209-fe83-4acb-9140-2e70b8f129b3-kube-api-access-9rrfh\") pod \"limitador-limitador-54d6d988d5-5b8ps\" (UID: \"e9e35209-fe83-4acb-9140-2e70b8f129b3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5b8ps" Jul 28 12:43:14.809883 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.809818 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-5b8ps" Jul 28 12:43:14.946505 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:14.946476 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:43:14.947747 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:43:14.947721 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e35209_fe83_4acb_9140_2e70b8f129b3.slice/crio-7b1fd77508d8dfcb42c801e0e6d1755c8f1abb3da889e40cc1dfc5b2e0cab300 WatchSource:0}: Error finding container 7b1fd77508d8dfcb42c801e0e6d1755c8f1abb3da889e40cc1dfc5b2e0cab300: Status 404 returned error can't find the container with id 7b1fd77508d8dfcb42c801e0e6d1755c8f1abb3da889e40cc1dfc5b2e0cab300 Jul 28 12:43:15.520826 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:15.520691 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-5b8ps" event={"ID":"e9e35209-fe83-4acb-9140-2e70b8f129b3","Type":"ContainerStarted","Data":"7b1fd77508d8dfcb42c801e0e6d1755c8f1abb3da889e40cc1dfc5b2e0cab300"} Jul 28 12:43:17.531850 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:17.531806 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-5b8ps" event={"ID":"e9e35209-fe83-4acb-9140-2e70b8f129b3","Type":"ContainerStarted","Data":"7503ba16525e4068b71924694e02de5a4de8d8bd8822ff1f2081b46e053e664e"} Jul 28 12:43:17.532280 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:17.531934 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-5b8ps" Jul 28 12:43:17.550227 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:17.550178 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-5b8ps" podStartSLOduration=1.183891628 podStartE2EDuration="3.550163322s" totalImagesPullingTime="2.366271694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:43:17.54813513 +0000 UTC m=+782.807425871" watchObservedRunningTime="2026-07-28 12:43:17.550163322 +0000 UTC m=+782.809454063" Jul 28 12:43:28.537479 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:43:28.537420 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-5b8ps" Jul 28 12:44:32.613094 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:32.613059 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:44:52.663627 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.663582 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7"] Jul 28 12:44:52.667214 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.667194 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.672192 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.672168 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 28 12:44:52.673956 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.673916 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 28 12:44:52.674042 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.673978 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-p75qd\"" Jul 28 12:44:52.674105 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.674091 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 28 12:44:52.689926 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.689893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.689926 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.689927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.690087 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.689951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.690087 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.690021 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfdtv\" (UniqueName: \"kubernetes.io/projected/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-kube-api-access-bfdtv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.690087 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.690056 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.690189 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.690100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.696470 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.696430 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7"] Jul 28 12:44:52.791177 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.791121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.791177 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.791196 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.791407 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.791226 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.791407 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.791244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.791407 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.791272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.791407 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.791315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfdtv\" (UniqueName: \"kubernetes.io/projected/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-kube-api-access-bfdtv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.791652 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.791627 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.791704 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.791657 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.791742 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.791707 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.793751 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.793732 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.793987 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.793968 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.804032 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.804004 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfdtv\" (UniqueName: \"kubernetes.io/projected/f90f3bdd-ec45-4ab6-bc41-363729f69bb1-kube-api-access-bfdtv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7\" (UID: \"f90f3bdd-ec45-4ab6-bc41-363729f69bb1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:52.977634 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:52.977547 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:44:53.131679 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:53.131653 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7"] Jul 28 12:44:53.132936 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:44:53.132910 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90f3bdd_ec45_4ab6_bc41_363729f69bb1.slice/crio-f14907fc1d954a440c93418c4b902ae4f56c0f5764480587bf55ab4d31e85362 WatchSource:0}: Error finding container f14907fc1d954a440c93418c4b902ae4f56c0f5764480587bf55ab4d31e85362: Status 404 returned error can't find the container with id f14907fc1d954a440c93418c4b902ae4f56c0f5764480587bf55ab4d31e85362 Jul 28 12:44:53.419089 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:53.419052 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:44:54.016752 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:54.016711 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" event={"ID":"f90f3bdd-ec45-4ab6-bc41-363729f69bb1","Type":"ContainerStarted","Data":"f14907fc1d954a440c93418c4b902ae4f56c0f5764480587bf55ab4d31e85362"} Jul 28 12:44:55.024765 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:55.024733 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" event={"ID":"f90f3bdd-ec45-4ab6-bc41-363729f69bb1","Type":"ContainerStarted","Data":"00179aadd15684d3fe9e9a1bea39445cba57f3e1e20921da60502ccb2d8a845e"} Jul 28 12:44:55.052246 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:44:55.052175 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" podStartSLOduration=1.854067012 podStartE2EDuration="3.052158951s" totalImagesPullingTime="1.198091939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:44:55.049571876 +0000 UTC m=+880.308862617" watchObservedRunningTime="2026-07-28 12:44:55.052158951 +0000 UTC m=+880.311449710" Jul 28 12:45:01.872059 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:01.872014 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:45:02.977996 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:02.977955 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:45:02.978393 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:02.978057 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:45:02.990383 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:02.990360 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:45:03.074868 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:03.074842 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7" Jul 28 12:45:15.149754 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:15.149723 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:45:22.947894 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:22.947859 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:45:34.789620 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:34.789582 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn"] Jul 28 12:45:34.793036 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:34.793017 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:34.805466 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:34.805422 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 28 12:45:34.822319 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:34.822286 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn"] Jul 28 12:45:34.929224 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:34.929187 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ab58fc0-b4b2-497f-bb79-07972d507846-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:34.929421 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:34.929244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ab58fc0-b4b2-497f-bb79-07972d507846-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:34.929421 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:34.929263 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab58fc0-b4b2-497f-bb79-07972d507846-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:34.929421 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:34.929334 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxq9x\" (UniqueName: \"kubernetes.io/projected/4ab58fc0-b4b2-497f-bb79-07972d507846-kube-api-access-lxq9x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:34.929421 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:34.929381 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab58fc0-b4b2-497f-bb79-07972d507846-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:34.929421 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:34.929405 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ab58fc0-b4b2-497f-bb79-07972d507846-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.030809 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.030766 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxq9x\" (UniqueName: \"kubernetes.io/projected/4ab58fc0-b4b2-497f-bb79-07972d507846-kube-api-access-lxq9x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.030809 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.030812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab58fc0-b4b2-497f-bb79-07972d507846-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.031072 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.030839 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ab58fc0-b4b2-497f-bb79-07972d507846-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.031072 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.030867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ab58fc0-b4b2-497f-bb79-07972d507846-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.031072 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.030911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ab58fc0-b4b2-497f-bb79-07972d507846-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.031072 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.030928 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab58fc0-b4b2-497f-bb79-07972d507846-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.031329 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.031302 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ab58fc0-b4b2-497f-bb79-07972d507846-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.031434 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.031364 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab58fc0-b4b2-497f-bb79-07972d507846-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.031510 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.031466 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ab58fc0-b4b2-497f-bb79-07972d507846-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.033288 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.033269 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ab58fc0-b4b2-497f-bb79-07972d507846-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.033560 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.033542 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab58fc0-b4b2-497f-bb79-07972d507846-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.047947 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.047890 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxq9x\" (UniqueName: \"kubernetes.io/projected/4ab58fc0-b4b2-497f-bb79-07972d507846-kube-api-access-lxq9x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sswpn\" (UID: \"4ab58fc0-b4b2-497f-bb79-07972d507846\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.103004 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.102968 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:35.266415 ip-10-0-136-145 kubenswrapper[2573]: W0728 12:45:35.266389 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab58fc0_b4b2_497f_bb79_07972d507846.slice/crio-259fcf446e2eb8d40b03fcb8070853c56f0589a591836d80b77d95ab7cdd8cb0 WatchSource:0}: Error finding container 259fcf446e2eb8d40b03fcb8070853c56f0589a591836d80b77d95ab7cdd8cb0: Status 404 returned error can't find the container with id 259fcf446e2eb8d40b03fcb8070853c56f0589a591836d80b77d95ab7cdd8cb0 Jul 28 12:45:35.269202 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:35.269181 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn"] Jul 28 12:45:36.231412 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:36.231372 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" event={"ID":"4ab58fc0-b4b2-497f-bb79-07972d507846","Type":"ContainerStarted","Data":"1419e39fad78cae0991f8a5f4aa5e1ce3ca1cb3cf3e6f39c31b3099f5563efeb"} Jul 28 12:45:36.231838 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:36.231463 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" event={"ID":"4ab58fc0-b4b2-497f-bb79-07972d507846","Type":"ContainerStarted","Data":"259fcf446e2eb8d40b03fcb8070853c56f0589a591836d80b77d95ab7cdd8cb0"} Jul 28 12:45:36.264330 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:36.264277 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" podStartSLOduration=1.703256477 podStartE2EDuration="2.264261882s" totalImagesPullingTime="561.005405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:45:36.260036512 +0000 UTC m=+921.519327253" watchObservedRunningTime="2026-07-28 12:45:36.264261882 +0000 UTC m=+921.523552649" Jul 28 12:45:36.978924 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:36.978892 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:45:45.103471 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:45.103419 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:45.103953 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:45.103486 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:45.116106 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:45.116079 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:45.287488 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:45.287456 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sswpn" Jul 28 12:45:50.072851 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:45:50.072814 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:47:03.942567 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:47:03.942533 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:47:13.352131 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:47:13.352100 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:47:25.267951 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:47:25.267919 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:49:09.150894 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:49:09.150859 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:49:18.849991 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:49:18.849955 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:49:48.246517 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:49:48.246429 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:49:57.949081 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:49:57.949046 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:50:08.442167 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:50:08.442131 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:50:14.138997 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:50:14.138958 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:50:24.741775 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:50:24.741735 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:50:34.845408 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:50:34.845373 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:50:46.042808 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:50:46.042775 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:50:56.452304 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:50:56.452270 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:51:18.646324 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:51:18.646244 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:51:30.747092 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:51:30.747052 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:51:55.763304 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:51:55.763271 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:52:06.361007 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:52:06.360971 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:52:16.958098 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:52:16.958054 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:52:33.260694 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:52:33.260653 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:52:42.461403 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:52:42.461310 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:52:53.151975 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:52:53.151937 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:53:02.049463 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:53:02.049404 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:53:11.146205 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:53:11.146162 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:53:23.149118 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:53:23.149086 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:53:31.674258 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:53:31.674221 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:55:12.561408 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:55:12.561335 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:55:23.236580 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:55:23.236545 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:55:38.337300 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:55:38.337220 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:55:54.937359 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:55:54.937309 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:56:07.430549 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:56:07.430514 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:56:15.441793 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:56:15.441749 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:56:26.132362 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:56:26.132326 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:56:36.940403 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:56:36.940366 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:56:46.255917 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:56:46.255883 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:56:57.638967 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:56:57.638929 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:57:35.127290 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:57:35.127204 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:57:45.140717 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:57:45.140685 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:57:57.461274 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:57:57.461237 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:58:16.830648 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:58:16.830616 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:58:35.229568 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:58:35.229534 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:58:47.730941 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:58:47.730852 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:58:58.232703 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:58:58.232669 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:59:10.326614 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:59:10.326578 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:59:26.966234 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:59:26.966202 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:59:34.958047 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:59:34.958001 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:59:50.148345 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:59:50.148311 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 12:59:59.638670 ip-10-0-136-145 kubenswrapper[2573]: I0728 12:59:59.638635 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:00:14.096285 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:00:14.096248 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:00:23.736411 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:00:23.736382 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:00:38.837239 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:00:38.837208 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:00:47.828107 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:00:47.828071 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:00:59.948787 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:00:59.948757 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:01:15.334528 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:01:15.334495 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:01:26.036697 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:01:26.036662 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:01:39.727680 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:01:39.727592 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:01:54.330338 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:01:54.330301 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:02:03.632170 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:02:03.632131 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:02:13.734507 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:02:13.734475 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:02:22.626894 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:02:22.626864 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:02:57.626313 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:02:57.626283 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:03:09.024339 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:09.024248 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:03:28.106408 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:28.106338 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq"] Jul 28 13:03:28.109797 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:28.109781 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq" Jul 28 13:03:28.112599 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:28.112577 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 13:03:28.116596 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:28.112999 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:03:28.120647 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:28.120220 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq"] Jul 28 13:03:28.138018 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:28.137989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59dee47-765e-421d-9365-7b0adbaea9a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq\" (UID: \"e59dee47-765e-421d-9365-7b0adbaea9a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq" Jul 28 13:03:28.238599 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:28.238573 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59dee47-765e-421d-9365-7b0adbaea9a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq\" (UID: \"e59dee47-765e-421d-9365-7b0adbaea9a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq" Jul 28 13:03:28.239183 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:28.239161 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59dee47-765e-421d-9365-7b0adbaea9a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq\" (UID: \"e59dee47-765e-421d-9365-7b0adbaea9a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq" Jul 28 13:03:28.426042 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:28.426015 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq" Jul 28 13:03:28.556405 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:28.556375 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq"] Jul 28 13:03:28.557980 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:03:28.557951 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59dee47_765e_421d_9365_7b0adbaea9a0.slice/crio-d9d7e63cb20b6a112d279d5b56a9c6e307e7513cd08ace16d09d01ced4418ac8 WatchSource:0}: Error finding container d9d7e63cb20b6a112d279d5b56a9c6e307e7513cd08ace16d09d01ced4418ac8: Status 404 returned error can't find the container with id d9d7e63cb20b6a112d279d5b56a9c6e307e7513cd08ace16d09d01ced4418ac8 Jul 28 13:03:28.559969 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:28.559951 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 13:03:28.679725 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:28.679656 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq" event={"ID":"e59dee47-765e-421d-9365-7b0adbaea9a0","Type":"ContainerStarted","Data":"d9d7e63cb20b6a112d279d5b56a9c6e307e7513cd08ace16d09d01ced4418ac8"} Jul 28 13:03:31.698889 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:31.698816 2573 generic.go:356] "Generic (PLEG): container finished" podID="e59dee47-765e-421d-9365-7b0adbaea9a0" containerID="2b59be5e1a153d9629e0706242253d41a0531e347c624f604318a00802b8b4ff" exitCode=0 Jul 28 13:03:31.698889 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:31.698866 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq" event={"ID":"e59dee47-765e-421d-9365-7b0adbaea9a0","Type":"ContainerDied","Data":"2b59be5e1a153d9629e0706242253d41a0531e347c624f604318a00802b8b4ff"} Jul 28 13:03:32.830926 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:32.830897 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq" Jul 28 13:03:32.875060 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:32.875034 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59dee47-765e-421d-9365-7b0adbaea9a0-maas-api-service-ca\") pod \"e59dee47-765e-421d-9365-7b0adbaea9a0\" (UID: \"e59dee47-765e-421d-9365-7b0adbaea9a0\") " Jul 28 13:03:32.875367 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:32.875344 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59dee47-765e-421d-9365-7b0adbaea9a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e59dee47-765e-421d-9365-7b0adbaea9a0" (UID: "e59dee47-765e-421d-9365-7b0adbaea9a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:03:32.975649 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:32.975582 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59dee47-765e-421d-9365-7b0adbaea9a0-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:03:33.710942 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:33.710915 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq" Jul 28 13:03:33.711108 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:33.710944 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq" event={"ID":"e59dee47-765e-421d-9365-7b0adbaea9a0","Type":"ContainerDied","Data":"d9d7e63cb20b6a112d279d5b56a9c6e307e7513cd08ace16d09d01ced4418ac8"} Jul 28 13:03:33.711108 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:33.710995 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9d7e63cb20b6a112d279d5b56a9c6e307e7513cd08ace16d09d01ced4418ac8" Jul 28 13:03:41.724240 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:41.724204 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:03:43.187563 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:43.187519 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq"] Jul 28 13:03:43.192642 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:43.192614 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-853a1b3f-nfmmq"] Jul 28 13:03:43.267848 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:43.267820 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e59dee47-765e-421d-9365-7b0adbaea9a0" path="/var/lib/kubelet/pods/e59dee47-765e-421d-9365-7b0adbaea9a0/volumes" Jul 28 13:03:53.332077 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:03:53.332040 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:04:12.060406 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.060371 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk"] Jul 28 13:04:12.060785 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.060697 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e59dee47-765e-421d-9365-7b0adbaea9a0" containerName="revoke-keys" Jul 28 13:04:12.060785 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.060707 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e59dee47-765e-421d-9365-7b0adbaea9a0" containerName="revoke-keys" Jul 28 13:04:12.060785 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.060767 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e59dee47-765e-421d-9365-7b0adbaea9a0" containerName="revoke-keys" Jul 28 13:04:12.068092 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.068078 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk" Jul 28 13:04:12.071369 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.071345 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk"] Jul 28 13:04:12.071570 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.071490 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 13:04:12.071709 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.071557 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:04:12.196787 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.196764 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852f1007-648e-4a9e-b1f2-7d6d075f56c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk\" (UID: \"852f1007-648e-4a9e-b1f2-7d6d075f56c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk" Jul 28 13:04:12.297687 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.297653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852f1007-648e-4a9e-b1f2-7d6d075f56c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk\" (UID: \"852f1007-648e-4a9e-b1f2-7d6d075f56c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk" Jul 28 13:04:12.298231 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.298212 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852f1007-648e-4a9e-b1f2-7d6d075f56c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk\" (UID: \"852f1007-648e-4a9e-b1f2-7d6d075f56c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk" Jul 28 13:04:12.379624 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.379566 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk" Jul 28 13:04:12.506167 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.506143 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk"] Jul 28 13:04:12.507655 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:04:12.507632 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852f1007_648e_4a9e_b1f2_7d6d075f56c6.slice/crio-0cff4ffc85c6c126db6c00f5405578565cdee8c6bfea3228b902f11fd5ac5fbe WatchSource:0}: Error finding container 0cff4ffc85c6c126db6c00f5405578565cdee8c6bfea3228b902f11fd5ac5fbe: Status 404 returned error can't find the container with id 0cff4ffc85c6c126db6c00f5405578565cdee8c6bfea3228b902f11fd5ac5fbe Jul 28 13:04:12.909390 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.909352 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk" event={"ID":"852f1007-648e-4a9e-b1f2-7d6d075f56c6","Type":"ContainerStarted","Data":"11757ee3aa3a255c89f5f5ee91d51fac2848184a07c0e73187e99453563b601f"} Jul 28 13:04:12.909587 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.909406 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk" event={"ID":"852f1007-648e-4a9e-b1f2-7d6d075f56c6","Type":"ContainerStarted","Data":"0cff4ffc85c6c126db6c00f5405578565cdee8c6bfea3228b902f11fd5ac5fbe"} Jul 28 13:04:12.929670 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:12.929623 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk" podStartSLOduration=0.92960558 podStartE2EDuration="929.60558ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:04:12.927876204 +0000 UTC m=+2038.187166943" watchObservedRunningTime="2026-07-28 13:04:12.92960558 +0000 UTC m=+2038.188896321" Jul 28 13:04:13.915522 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:13.915480 2573 generic.go:356] "Generic (PLEG): container finished" podID="852f1007-648e-4a9e-b1f2-7d6d075f56c6" containerID="11757ee3aa3a255c89f5f5ee91d51fac2848184a07c0e73187e99453563b601f" exitCode=0 Jul 28 13:04:13.915942 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:13.915568 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk" event={"ID":"852f1007-648e-4a9e-b1f2-7d6d075f56c6","Type":"ContainerDied","Data":"11757ee3aa3a255c89f5f5ee91d51fac2848184a07c0e73187e99453563b601f"} Jul 28 13:04:15.045581 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:15.045558 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk" Jul 28 13:04:15.119333 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:15.119309 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852f1007-648e-4a9e-b1f2-7d6d075f56c6-maas-api-service-ca\") pod \"852f1007-648e-4a9e-b1f2-7d6d075f56c6\" (UID: \"852f1007-648e-4a9e-b1f2-7d6d075f56c6\") " Jul 28 13:04:15.119608 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:15.119590 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852f1007-648e-4a9e-b1f2-7d6d075f56c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "852f1007-648e-4a9e-b1f2-7d6d075f56c6" (UID: "852f1007-648e-4a9e-b1f2-7d6d075f56c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:04:15.220328 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:15.220269 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852f1007-648e-4a9e-b1f2-7d6d075f56c6-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:04:15.927016 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:15.926984 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk" event={"ID":"852f1007-648e-4a9e-b1f2-7d6d075f56c6","Type":"ContainerDied","Data":"0cff4ffc85c6c126db6c00f5405578565cdee8c6bfea3228b902f11fd5ac5fbe"} Jul 28 13:04:15.927016 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:15.927002 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk" Jul 28 13:04:15.927215 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:15.927055 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cff4ffc85c6c126db6c00f5405578565cdee8c6bfea3228b902f11fd5ac5fbe" Jul 28 13:04:25.925161 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:25.925127 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:04:27.158117 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:27.158079 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk"] Jul 28 13:04:27.166239 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:27.166211 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce27c48c-5z5qk"] Jul 28 13:04:27.267518 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:27.267483 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="852f1007-648e-4a9e-b1f2-7d6d075f56c6" path="/var/lib/kubelet/pods/852f1007-648e-4a9e-b1f2-7d6d075f56c6/volumes" Jul 28 13:04:36.834032 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:36.833948 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:04:56.461364 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:56.461331 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f"] Jul 28 13:04:56.462040 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:56.461696 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="852f1007-648e-4a9e-b1f2-7d6d075f56c6" containerName="revoke-keys" Jul 28 13:04:56.462040 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:56.461708 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="852f1007-648e-4a9e-b1f2-7d6d075f56c6" containerName="revoke-keys" Jul 28 13:04:56.462040 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:56.461773 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="852f1007-648e-4a9e-b1f2-7d6d075f56c6" containerName="revoke-keys" Jul 28 13:04:56.464611 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:56.464595 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f" Jul 28 13:04:56.467605 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:56.467583 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:04:56.467715 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:56.467629 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 13:04:56.474637 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:56.474616 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f"] Jul 28 13:04:56.555778 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:56.555747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7ea2ca-e633-47df-854b-b6cda902619e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f\" (UID: \"5c7ea2ca-e633-47df-854b-b6cda902619e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f" Jul 28 13:04:56.656900 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:56.656872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7ea2ca-e633-47df-854b-b6cda902619e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f\" (UID: \"5c7ea2ca-e633-47df-854b-b6cda902619e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f" Jul 28 13:04:56.657490 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:56.657474 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7ea2ca-e633-47df-854b-b6cda902619e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f\" (UID: \"5c7ea2ca-e633-47df-854b-b6cda902619e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f" Jul 28 13:04:56.775671 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:56.775616 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f" Jul 28 13:04:56.906620 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:56.906588 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f"] Jul 28 13:04:56.907666 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:04:56.907631 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7ea2ca_e633_47df_854b_b6cda902619e.slice/crio-0d35a17cda3fe33ed612d46c08f09c68caa93c6457d6adf095eaf714790192e4 WatchSource:0}: Error finding container 0d35a17cda3fe33ed612d46c08f09c68caa93c6457d6adf095eaf714790192e4: Status 404 returned error can't find the container with id 0d35a17cda3fe33ed612d46c08f09c68caa93c6457d6adf095eaf714790192e4 Jul 28 13:04:57.142682 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:57.142653 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f" event={"ID":"5c7ea2ca-e633-47df-854b-b6cda902619e","Type":"ContainerStarted","Data":"0d35a17cda3fe33ed612d46c08f09c68caa93c6457d6adf095eaf714790192e4"} Jul 28 13:04:58.149514 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:58.149479 2573 generic.go:356] "Generic (PLEG): container finished" podID="5c7ea2ca-e633-47df-854b-b6cda902619e" containerID="67cfb3d70cba7bd032e65e911e42590c6c0f70a57003ee1152e474f2737067e5" exitCode=0 Jul 28 13:04:58.149932 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:58.149564 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f" event={"ID":"5c7ea2ca-e633-47df-854b-b6cda902619e","Type":"ContainerDied","Data":"67cfb3d70cba7bd032e65e911e42590c6c0f70a57003ee1152e474f2737067e5"} Jul 28 13:04:59.283026 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:59.283000 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f" Jul 28 13:04:59.384151 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:59.384124 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7ea2ca-e633-47df-854b-b6cda902619e-maas-api-service-ca\") pod \"5c7ea2ca-e633-47df-854b-b6cda902619e\" (UID: \"5c7ea2ca-e633-47df-854b-b6cda902619e\") " Jul 28 13:04:59.384552 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:59.384519 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7ea2ca-e633-47df-854b-b6cda902619e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c7ea2ca-e633-47df-854b-b6cda902619e" (UID: "5c7ea2ca-e633-47df-854b-b6cda902619e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:04:59.485134 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:04:59.485073 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7ea2ca-e633-47df-854b-b6cda902619e-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:05:00.162154 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:05:00.162129 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f" Jul 28 13:05:00.162331 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:05:00.162125 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f" event={"ID":"5c7ea2ca-e633-47df-854b-b6cda902619e","Type":"ContainerDied","Data":"0d35a17cda3fe33ed612d46c08f09c68caa93c6457d6adf095eaf714790192e4"} Jul 28 13:05:00.162331 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:05:00.162250 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d35a17cda3fe33ed612d46c08f09c68caa93c6457d6adf095eaf714790192e4" Jul 28 13:05:10.227335 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:05:10.227292 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:05:11.537261 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:05:11.537222 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f"] Jul 28 13:05:11.540743 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:05:11.540717 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd87d0c9-62j9f"] Jul 28 13:05:13.268890 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:05:13.268856 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c7ea2ca-e633-47df-854b-b6cda902619e" path="/var/lib/kubelet/pods/5c7ea2ca-e633-47df-854b-b6cda902619e/volumes" Jul 28 13:06:38.032127 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:06:38.032043 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:06:54.131572 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:06:54.131544 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:07:03.732908 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:03.732871 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:07:12.328784 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:12.328747 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:07:19.106610 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.106573 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp"] Jul 28 13:07:19.106977 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.106920 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c7ea2ca-e633-47df-854b-b6cda902619e" containerName="revoke-keys" Jul 28 13:07:19.106977 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.106932 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c7ea2ca-e633-47df-854b-b6cda902619e" containerName="revoke-keys" Jul 28 13:07:19.107049 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.107028 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c7ea2ca-e633-47df-854b-b6cda902619e" containerName="revoke-keys" Jul 28 13:07:19.109993 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.109974 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp" Jul 28 13:07:19.113419 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.113394 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:07:19.114356 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.114339 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 13:07:19.123174 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.123152 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp"] Jul 28 13:07:19.129880 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.129848 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93254b8a-39ff-4933-b906-ebf49fa96b24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp\" (UID: \"93254b8a-39ff-4933-b906-ebf49fa96b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp" Jul 28 13:07:19.230896 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.230870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93254b8a-39ff-4933-b906-ebf49fa96b24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp\" (UID: \"93254b8a-39ff-4933-b906-ebf49fa96b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp" Jul 28 13:07:19.231423 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.231407 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93254b8a-39ff-4933-b906-ebf49fa96b24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp\" (UID: \"93254b8a-39ff-4933-b906-ebf49fa96b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp" Jul 28 13:07:19.420105 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.420084 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp" Jul 28 13:07:19.556816 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.556787 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp"] Jul 28 13:07:19.558253 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:07:19.558229 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93254b8a_39ff_4933_b906_ebf49fa96b24.slice/crio-1056e469d20176628a743e3f60e8063ad6b2843efafbb35ef1551c7769bd3c6b WatchSource:0}: Error finding container 1056e469d20176628a743e3f60e8063ad6b2843efafbb35ef1551c7769bd3c6b: Status 404 returned error can't find the container with id 1056e469d20176628a743e3f60e8063ad6b2843efafbb35ef1551c7769bd3c6b Jul 28 13:07:19.884921 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.884884 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp" event={"ID":"93254b8a-39ff-4933-b906-ebf49fa96b24","Type":"ContainerStarted","Data":"a8a424122cc21c8335285eac7ae73b9ecdbcfbcf37f3c8f2b956429c1c3c7c4c"} Jul 28 13:07:19.885063 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.884942 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp" event={"ID":"93254b8a-39ff-4933-b906-ebf49fa96b24","Type":"ContainerStarted","Data":"1056e469d20176628a743e3f60e8063ad6b2843efafbb35ef1551c7769bd3c6b"} Jul 28 13:07:19.903333 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:19.903291 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp" podStartSLOduration=0.903277734 podStartE2EDuration="903.277734ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:07:19.901053149 +0000 UTC m=+2225.160343893" watchObservedRunningTime="2026-07-28 13:07:19.903277734 +0000 UTC m=+2225.162568474" Jul 28 13:07:20.891122 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:20.891087 2573 generic.go:356] "Generic (PLEG): container finished" podID="93254b8a-39ff-4933-b906-ebf49fa96b24" containerID="a8a424122cc21c8335285eac7ae73b9ecdbcfbcf37f3c8f2b956429c1c3c7c4c" exitCode=0 Jul 28 13:07:20.891573 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:20.891155 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp" event={"ID":"93254b8a-39ff-4933-b906-ebf49fa96b24","Type":"ContainerDied","Data":"a8a424122cc21c8335285eac7ae73b9ecdbcfbcf37f3c8f2b956429c1c3c7c4c"} Jul 28 13:07:22.024729 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:22.024705 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp" Jul 28 13:07:22.052749 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:22.052727 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93254b8a-39ff-4933-b906-ebf49fa96b24-maas-api-service-ca\") pod \"93254b8a-39ff-4933-b906-ebf49fa96b24\" (UID: \"93254b8a-39ff-4933-b906-ebf49fa96b24\") " Jul 28 13:07:22.053131 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:22.053105 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93254b8a-39ff-4933-b906-ebf49fa96b24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93254b8a-39ff-4933-b906-ebf49fa96b24" (UID: "93254b8a-39ff-4933-b906-ebf49fa96b24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:07:22.154137 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:22.154078 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93254b8a-39ff-4933-b906-ebf49fa96b24-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:07:22.902240 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:22.902211 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp" Jul 28 13:07:22.902422 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:22.902202 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp" event={"ID":"93254b8a-39ff-4933-b906-ebf49fa96b24","Type":"ContainerDied","Data":"1056e469d20176628a743e3f60e8063ad6b2843efafbb35ef1551c7769bd3c6b"} Jul 28 13:07:22.902422 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:22.902348 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1056e469d20176628a743e3f60e8063ad6b2843efafbb35ef1551c7769bd3c6b" Jul 28 13:07:32.529770 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:32.529735 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:07:34.170684 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:34.170652 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp"] Jul 28 13:07:34.173737 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:34.173713 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c499a1e4-txnbp"] Jul 28 13:07:35.267665 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:07:35.267636 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93254b8a-39ff-4933-b906-ebf49fa96b24" path="/var/lib/kubelet/pods/93254b8a-39ff-4933-b906-ebf49fa96b24/volumes" Jul 28 13:08:10.625241 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:10.625208 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:08:19.425585 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:19.425553 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:08:33.627424 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:33.627395 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:08:38.478576 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:38.478539 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk"] Jul 28 13:08:38.479065 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:38.479045 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93254b8a-39ff-4933-b906-ebf49fa96b24" containerName="revoke-keys" Jul 28 13:08:38.479142 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:38.479068 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="93254b8a-39ff-4933-b906-ebf49fa96b24" containerName="revoke-keys" Jul 28 13:08:38.479194 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:38.479153 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="93254b8a-39ff-4933-b906-ebf49fa96b24" containerName="revoke-keys" Jul 28 13:08:38.482099 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:38.482080 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk" Jul 28 13:08:38.484873 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:38.484852 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 13:08:38.484967 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:38.484910 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:08:38.490037 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:38.489846 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk"] Jul 28 13:08:38.565625 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:38.565602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c2830aa-8bbd-457b-aa01-bd836f8617a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk\" (UID: \"3c2830aa-8bbd-457b-aa01-bd836f8617a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk" Jul 28 13:08:38.666222 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:38.666193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c2830aa-8bbd-457b-aa01-bd836f8617a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk\" (UID: \"3c2830aa-8bbd-457b-aa01-bd836f8617a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk" Jul 28 13:08:38.666806 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:38.666788 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c2830aa-8bbd-457b-aa01-bd836f8617a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk\" (UID: \"3c2830aa-8bbd-457b-aa01-bd836f8617a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk" Jul 28 13:08:38.794156 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:38.794086 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk" Jul 28 13:08:38.927293 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:38.927266 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk"] Jul 28 13:08:38.928673 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:08:38.928644 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2830aa_8bbd_457b_aa01_bd836f8617a2.slice/crio-685b4782efc65288360afeee4eae79e4849f1dc7db55f40dacf6a27952fdc092 WatchSource:0}: Error finding container 685b4782efc65288360afeee4eae79e4849f1dc7db55f40dacf6a27952fdc092: Status 404 returned error can't find the container with id 685b4782efc65288360afeee4eae79e4849f1dc7db55f40dacf6a27952fdc092 Jul 28 13:08:39.295409 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:39.295315 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk" event={"ID":"3c2830aa-8bbd-457b-aa01-bd836f8617a2","Type":"ContainerStarted","Data":"634474b09a39323e12790768566516a9b2368b2f0f8edd26c6cc7088555e4ff8"} Jul 28 13:08:39.295409 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:39.295356 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk" event={"ID":"3c2830aa-8bbd-457b-aa01-bd836f8617a2","Type":"ContainerStarted","Data":"685b4782efc65288360afeee4eae79e4849f1dc7db55f40dacf6a27952fdc092"} Jul 28 13:08:39.313543 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:39.313498 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk" podStartSLOduration=1.3134831710000001 podStartE2EDuration="1.313483171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:08:39.310270125 +0000 UTC m=+2304.569560868" watchObservedRunningTime="2026-07-28 13:08:39.313483171 +0000 UTC m=+2304.572773908" Jul 28 13:08:40.302236 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:40.302201 2573 generic.go:356] "Generic (PLEG): container finished" podID="3c2830aa-8bbd-457b-aa01-bd836f8617a2" containerID="634474b09a39323e12790768566516a9b2368b2f0f8edd26c6cc7088555e4ff8" exitCode=0 Jul 28 13:08:40.302236 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:40.302237 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk" event={"ID":"3c2830aa-8bbd-457b-aa01-bd836f8617a2","Type":"ContainerDied","Data":"634474b09a39323e12790768566516a9b2368b2f0f8edd26c6cc7088555e4ff8"} Jul 28 13:08:41.435872 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:41.435851 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk" Jul 28 13:08:41.490824 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:41.490798 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c2830aa-8bbd-457b-aa01-bd836f8617a2-maas-api-service-ca\") pod \"3c2830aa-8bbd-457b-aa01-bd836f8617a2\" (UID: \"3c2830aa-8bbd-457b-aa01-bd836f8617a2\") " Jul 28 13:08:41.491115 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:41.491094 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2830aa-8bbd-457b-aa01-bd836f8617a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c2830aa-8bbd-457b-aa01-bd836f8617a2" (UID: "3c2830aa-8bbd-457b-aa01-bd836f8617a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:08:41.591392 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:41.591337 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c2830aa-8bbd-457b-aa01-bd836f8617a2-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:08:42.312983 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:42.312946 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk" Jul 28 13:08:42.313224 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:42.312982 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk" event={"ID":"3c2830aa-8bbd-457b-aa01-bd836f8617a2","Type":"ContainerDied","Data":"685b4782efc65288360afeee4eae79e4849f1dc7db55f40dacf6a27952fdc092"} Jul 28 13:08:42.313224 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:42.313029 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="685b4782efc65288360afeee4eae79e4849f1dc7db55f40dacf6a27952fdc092" Jul 28 13:08:52.523680 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:52.523642 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:08:58.546542 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:58.546507 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk"] Jul 28 13:08:58.549410 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:58.549385 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0e7d2b19-glhrk"] Jul 28 13:08:59.267623 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:08:59.267590 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c2830aa-8bbd-457b-aa01-bd836f8617a2" path="/var/lib/kubelet/pods/3c2830aa-8bbd-457b-aa01-bd836f8617a2/volumes" Jul 28 13:09:01.323694 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:09:01.323664 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:09:54.433515 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:09:54.433474 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:10:06.639609 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:10:06.639573 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:10:15.391100 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:10:15.391072 2573 scope.go:117] "RemoveContainer" containerID="11757ee3aa3a255c89f5f5ee91d51fac2848184a07c0e73187e99453563b601f" Jul 28 13:10:15.399387 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:10:15.399368 2573 scope.go:117] "RemoveContainer" containerID="2b59be5e1a153d9629e0706242253d41a0531e347c624f604318a00802b8b4ff" Jul 28 13:10:57.023281 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:10:57.023249 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:11:08.141332 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:08.141296 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:11:15.423726 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:15.423698 2573 scope.go:117] "RemoveContainer" containerID="67cfb3d70cba7bd032e65e911e42590c6c0f70a57003ee1152e474f2737067e5" Jul 28 13:11:16.326030 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:16.325991 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:11:28.125054 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:28.125018 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:11:36.022659 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.022628 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh"] Jul 28 13:11:36.023076 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.022970 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c2830aa-8bbd-457b-aa01-bd836f8617a2" containerName="revoke-keys" Jul 28 13:11:36.023076 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.022989 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c2830aa-8bbd-457b-aa01-bd836f8617a2" containerName="revoke-keys" Jul 28 13:11:36.023076 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.023043 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c2830aa-8bbd-457b-aa01-bd836f8617a2" containerName="revoke-keys" Jul 28 13:11:36.026200 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.026172 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.029274 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.029256 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e75cd7\"/\"kube-root-ca.crt\"" Jul 28 13:11:36.030709 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.030677 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e75cd7\"/\"test-model-e75cd7-kserve-self-signed-certs\"" Jul 28 13:11:36.030709 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.030700 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e75cd7\"/\"default-dockercfg-t9jzp\"" Jul 28 13:11:36.030836 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.030683 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e75cd7\"/\"openshift-service-ca.crt\"" Jul 28 13:11:36.036664 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.036640 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh"] Jul 28 13:11:36.055432 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.055399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshzk\" (UniqueName: \"kubernetes.io/projected/e0d21718-223b-4bac-94da-23452e54e260-kube-api-access-cshzk\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.055560 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.055454 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-model-cache\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.055560 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.055515 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d21718-223b-4bac-94da-23452e54e260-tls-certs\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.055654 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.055612 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-dshm\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.055704 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.055664 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-home\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.055749 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.055691 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-tmp-dir\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.156685 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.156648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-home\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.156863 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.156693 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-tmp-dir\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.156863 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.156717 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshzk\" (UniqueName: \"kubernetes.io/projected/e0d21718-223b-4bac-94da-23452e54e260-kube-api-access-cshzk\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.156863 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.156833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-model-cache\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.157029 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.156879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d21718-223b-4bac-94da-23452e54e260-tls-certs\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.157029 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.156945 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-dshm\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.157140 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.157098 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-tmp-dir\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.157196 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.157157 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-home\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.157196 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.157169 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-model-cache\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.159199 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.159179 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-dshm\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.159601 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.159579 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d21718-223b-4bac-94da-23452e54e260-tls-certs\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.166133 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.166088 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshzk\" (UniqueName: \"kubernetes.io/projected/e0d21718-223b-4bac-94da-23452e54e260-kube-api-access-cshzk\") pod \"test-model-e75cd7-kserve-5b45dc9ff-7cbgh\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.335397 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.335304 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:36.469758 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:36.469686 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh"] Jul 28 13:11:36.472071 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:11:36.472048 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0d21718_223b_4bac_94da_23452e54e260.slice/crio-b56093ba56a15e1341ef4a2e3b73052baab7d31773284f906fa9d4983e282351 WatchSource:0}: Error finding container b56093ba56a15e1341ef4a2e3b73052baab7d31773284f906fa9d4983e282351: Status 404 returned error can't find the container with id b56093ba56a15e1341ef4a2e3b73052baab7d31773284f906fa9d4983e282351 Jul 28 13:11:37.215263 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:37.215224 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" event={"ID":"e0d21718-223b-4bac-94da-23452e54e260","Type":"ContainerStarted","Data":"e3837f0b730fe8d59e1fe547215901044263baa85e0e62b216d8f83f3f160979"} Jul 28 13:11:37.215644 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:37.215273 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" event={"ID":"e0d21718-223b-4bac-94da-23452e54e260","Type":"ContainerStarted","Data":"b56093ba56a15e1341ef4a2e3b73052baab7d31773284f906fa9d4983e282351"} Jul 28 13:11:37.234264 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:37.234216 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" podStartSLOduration=1.2342015 podStartE2EDuration="1.2342015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:11:37.233347758 +0000 UTC m=+2482.492638503" watchObservedRunningTime="2026-07-28 13:11:37.2342015 +0000 UTC m=+2482.493492239" Jul 28 13:11:39.732583 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:39.732551 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:11:46.336136 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:46.336102 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:46.336525 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:46.336148 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:46.348091 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:46.348066 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:47.280939 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:47.280913 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:11:55.027814 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:11:55.027780 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:12:22.342981 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:22.342946 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh"] Jul 28 13:12:22.501280 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:22.501250 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb"] Jul 28 13:12:22.504473 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:22.504454 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb" Jul 28 13:12:22.507322 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:22.507298 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:12:22.507415 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:22.507340 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 13:12:22.513139 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:22.513117 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb"] Jul 28 13:12:22.627534 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:22.627462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a96d57-96a2-4ad8-8b3d-78c285c264d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb\" (UID: \"95a96d57-96a2-4ad8-8b3d-78c285c264d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb" Jul 28 13:12:22.728217 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:22.728186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a96d57-96a2-4ad8-8b3d-78c285c264d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb\" (UID: \"95a96d57-96a2-4ad8-8b3d-78c285c264d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb" Jul 28 13:12:22.728841 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:22.728822 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a96d57-96a2-4ad8-8b3d-78c285c264d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb\" (UID: \"95a96d57-96a2-4ad8-8b3d-78c285c264d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb" Jul 28 13:12:22.815286 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:22.815256 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb" Jul 28 13:12:22.945901 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:22.945875 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb"] Jul 28 13:12:22.947193 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:12:22.947164 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a96d57_96a2_4ad8_8b3d_78c285c264d0.slice/crio-e3358db89b46c86ffbb926edbcf2d7c1609ce7a9dc50d8a0256aead3d1b5afc0 WatchSource:0}: Error finding container e3358db89b46c86ffbb926edbcf2d7c1609ce7a9dc50d8a0256aead3d1b5afc0: Status 404 returned error can't find the container with id e3358db89b46c86ffbb926edbcf2d7c1609ce7a9dc50d8a0256aead3d1b5afc0 Jul 28 13:12:23.453560 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:23.453518 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb" event={"ID":"95a96d57-96a2-4ad8-8b3d-78c285c264d0","Type":"ContainerStarted","Data":"410e0c6ac8d6060b73919f03ff3347322a58fd09c06689a57b66832d870d2856"} Jul 28 13:12:23.453932 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:23.453578 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb" event={"ID":"95a96d57-96a2-4ad8-8b3d-78c285c264d0","Type":"ContainerStarted","Data":"e3358db89b46c86ffbb926edbcf2d7c1609ce7a9dc50d8a0256aead3d1b5afc0"} Jul 28 13:12:23.486363 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:23.486312 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb" podStartSLOduration=1.486297992 podStartE2EDuration="1.486297992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:12:23.484404064 +0000 UTC m=+2528.743694806" watchObservedRunningTime="2026-07-28 13:12:23.486297992 +0000 UTC m=+2528.745588733" Jul 28 13:12:24.460422 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:24.460380 2573 generic.go:356] "Generic (PLEG): container finished" podID="95a96d57-96a2-4ad8-8b3d-78c285c264d0" containerID="410e0c6ac8d6060b73919f03ff3347322a58fd09c06689a57b66832d870d2856" exitCode=0 Jul 28 13:12:24.460808 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:24.460459 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb" event={"ID":"95a96d57-96a2-4ad8-8b3d-78c285c264d0","Type":"ContainerDied","Data":"410e0c6ac8d6060b73919f03ff3347322a58fd09c06689a57b66832d870d2856"} Jul 28 13:12:25.593564 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:25.593539 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb" Jul 28 13:12:25.753803 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:25.753722 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a96d57-96a2-4ad8-8b3d-78c285c264d0-maas-api-service-ca\") pod \"95a96d57-96a2-4ad8-8b3d-78c285c264d0\" (UID: \"95a96d57-96a2-4ad8-8b3d-78c285c264d0\") " Jul 28 13:12:25.754013 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:25.753991 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a96d57-96a2-4ad8-8b3d-78c285c264d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95a96d57-96a2-4ad8-8b3d-78c285c264d0" (UID: "95a96d57-96a2-4ad8-8b3d-78c285c264d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:12:25.854391 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:25.854360 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a96d57-96a2-4ad8-8b3d-78c285c264d0-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:12:26.035111 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:26.034996 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:12:26.471764 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:26.471734 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb" Jul 28 13:12:26.471764 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:26.471749 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb" event={"ID":"95a96d57-96a2-4ad8-8b3d-78c285c264d0","Type":"ContainerDied","Data":"e3358db89b46c86ffbb926edbcf2d7c1609ce7a9dc50d8a0256aead3d1b5afc0"} Jul 28 13:12:26.471968 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:26.471803 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3358db89b46c86ffbb926edbcf2d7c1609ce7a9dc50d8a0256aead3d1b5afc0" Jul 28 13:12:37.362126 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.362084 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" podUID="e0d21718-223b-4bac-94da-23452e54e260" containerName="main" containerID="cri-o://e3837f0b730fe8d59e1fe547215901044263baa85e0e62b216d8f83f3f160979" gracePeriod=45 Jul 28 13:12:37.532081 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.532052 2573 generic.go:356] "Generic (PLEG): container finished" podID="e0d21718-223b-4bac-94da-23452e54e260" containerID="e3837f0b730fe8d59e1fe547215901044263baa85e0e62b216d8f83f3f160979" exitCode=0 Jul 28 13:12:37.532230 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.532130 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" event={"ID":"e0d21718-223b-4bac-94da-23452e54e260","Type":"ContainerDied","Data":"e3837f0b730fe8d59e1fe547215901044263baa85e0e62b216d8f83f3f160979"} Jul 28 13:12:37.576035 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.576004 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb"] Jul 28 13:12:37.580391 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.580363 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c260d4-8xnnb"] Jul 28 13:12:37.616994 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.616944 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:12:37.655134 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.655108 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-model-cache\") pod \"e0d21718-223b-4bac-94da-23452e54e260\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " Jul 28 13:12:37.655235 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.655138 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-dshm\") pod \"e0d21718-223b-4bac-94da-23452e54e260\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " Jul 28 13:12:37.655235 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.655176 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d21718-223b-4bac-94da-23452e54e260-tls-certs\") pod \"e0d21718-223b-4bac-94da-23452e54e260\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " Jul 28 13:12:37.655235 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.655204 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-tmp-dir\") pod \"e0d21718-223b-4bac-94da-23452e54e260\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " Jul 28 13:12:37.655235 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.655224 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cshzk\" (UniqueName: \"kubernetes.io/projected/e0d21718-223b-4bac-94da-23452e54e260-kube-api-access-cshzk\") pod \"e0d21718-223b-4bac-94da-23452e54e260\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " Jul 28 13:12:37.655482 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.655252 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-home\") pod \"e0d21718-223b-4bac-94da-23452e54e260\" (UID: \"e0d21718-223b-4bac-94da-23452e54e260\") " Jul 28 13:12:37.655482 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.655368 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0d21718-223b-4bac-94da-23452e54e260" (UID: "e0d21718-223b-4bac-94da-23452e54e260"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:12:37.655592 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.655544 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-model-cache\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:12:37.655592 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.655513 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0d21718-223b-4bac-94da-23452e54e260" (UID: "e0d21718-223b-4bac-94da-23452e54e260"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:12:37.655672 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.655589 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-home" (OuterVolumeSpecName: "home") pod "e0d21718-223b-4bac-94da-23452e54e260" (UID: "e0d21718-223b-4bac-94da-23452e54e260"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:12:37.657511 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.657486 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-dshm" (OuterVolumeSpecName: "dshm") pod "e0d21718-223b-4bac-94da-23452e54e260" (UID: "e0d21718-223b-4bac-94da-23452e54e260"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:12:37.657601 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.657510 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d21718-223b-4bac-94da-23452e54e260-kube-api-access-cshzk" (OuterVolumeSpecName: "kube-api-access-cshzk") pod "e0d21718-223b-4bac-94da-23452e54e260" (UID: "e0d21718-223b-4bac-94da-23452e54e260"). InnerVolumeSpecName "kube-api-access-cshzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:12:37.657601 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.657584 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d21718-223b-4bac-94da-23452e54e260-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0d21718-223b-4bac-94da-23452e54e260" (UID: "e0d21718-223b-4bac-94da-23452e54e260"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:12:37.756913 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.756885 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-dshm\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:12:37.756913 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.756909 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d21718-223b-4bac-94da-23452e54e260-tls-certs\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:12:37.756913 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.756920 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-tmp-dir\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:12:37.757133 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.756930 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cshzk\" (UniqueName: \"kubernetes.io/projected/e0d21718-223b-4bac-94da-23452e54e260-kube-api-access-cshzk\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:12:37.757133 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:37.756962 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d21718-223b-4bac-94da-23452e54e260-home\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:12:38.539107 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:38.539077 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" Jul 28 13:12:38.539107 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:38.539093 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh" event={"ID":"e0d21718-223b-4bac-94da-23452e54e260","Type":"ContainerDied","Data":"b56093ba56a15e1341ef4a2e3b73052baab7d31773284f906fa9d4983e282351"} Jul 28 13:12:38.539634 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:38.539138 2573 scope.go:117] "RemoveContainer" containerID="e3837f0b730fe8d59e1fe547215901044263baa85e0e62b216d8f83f3f160979" Jul 28 13:12:38.564254 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:38.564225 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh"] Jul 28 13:12:38.568023 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:38.568003 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e75cd7/test-model-e75cd7-kserve-5b45dc9ff-7cbgh"] Jul 28 13:12:39.267726 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:39.267691 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95a96d57-96a2-4ad8-8b3d-78c285c264d0" path="/var/lib/kubelet/pods/95a96d57-96a2-4ad8-8b3d-78c285c264d0/volumes" Jul 28 13:12:39.268084 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:39.268069 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0d21718-223b-4bac-94da-23452e54e260" path="/var/lib/kubelet/pods/e0d21718-223b-4bac-94da-23452e54e260/volumes" Jul 28 13:12:42.430827 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:42.430792 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:12:44.471641 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.471607 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp"] Jul 28 13:12:44.472055 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.471945 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d21718-223b-4bac-94da-23452e54e260" containerName="main" Jul 28 13:12:44.472055 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.471955 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d21718-223b-4bac-94da-23452e54e260" containerName="main" Jul 28 13:12:44.472055 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.471977 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a96d57-96a2-4ad8-8b3d-78c285c264d0" containerName="revoke-keys" Jul 28 13:12:44.472055 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.471982 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a96d57-96a2-4ad8-8b3d-78c285c264d0" containerName="revoke-keys" Jul 28 13:12:44.472055 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.472038 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d21718-223b-4bac-94da-23452e54e260" containerName="main" Jul 28 13:12:44.472055 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.472047 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a96d57-96a2-4ad8-8b3d-78c285c264d0" containerName="revoke-keys" Jul 28 13:12:44.476238 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.476220 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp" Jul 28 13:12:44.479282 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.479262 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 13:12:44.479419 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.479360 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:12:44.483965 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.483719 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp"] Jul 28 13:12:44.511281 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.511242 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bb46bc-3b2f-483f-99ba-5bee6e6b564b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp\" (UID: \"53bb46bc-3b2f-483f-99ba-5bee6e6b564b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp" Jul 28 13:12:44.611967 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.611929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bb46bc-3b2f-483f-99ba-5bee6e6b564b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp\" (UID: \"53bb46bc-3b2f-483f-99ba-5bee6e6b564b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp" Jul 28 13:12:44.612601 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.612579 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bb46bc-3b2f-483f-99ba-5bee6e6b564b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp\" (UID: \"53bb46bc-3b2f-483f-99ba-5bee6e6b564b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp" Jul 28 13:12:44.788479 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.788381 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp" Jul 28 13:12:44.920979 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:44.920950 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp"] Jul 28 13:12:44.922422 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:12:44.922398 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bb46bc_3b2f_483f_99ba_5bee6e6b564b.slice/crio-b2ff9be6d90ba73a16d633e52e0871c4c84ea41d5e7d61bef2066c69fb5ca192 WatchSource:0}: Error finding container b2ff9be6d90ba73a16d633e52e0871c4c84ea41d5e7d61bef2066c69fb5ca192: Status 404 returned error can't find the container with id b2ff9be6d90ba73a16d633e52e0871c4c84ea41d5e7d61bef2066c69fb5ca192 Jul 28 13:12:45.576618 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:45.576581 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp" event={"ID":"53bb46bc-3b2f-483f-99ba-5bee6e6b564b","Type":"ContainerStarted","Data":"b2d4f482a21d1c3fb9e3f65638d36501056c9d39262bc8c33fa8682b9c110fe8"} Jul 28 13:12:45.577077 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:45.576674 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp" event={"ID":"53bb46bc-3b2f-483f-99ba-5bee6e6b564b","Type":"ContainerStarted","Data":"b2ff9be6d90ba73a16d633e52e0871c4c84ea41d5e7d61bef2066c69fb5ca192"} Jul 28 13:12:45.594603 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:45.594563 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp" podStartSLOduration=1.594548957 podStartE2EDuration="1.594548957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:12:45.59270586 +0000 UTC m=+2550.851996601" watchObservedRunningTime="2026-07-28 13:12:45.594548957 +0000 UTC m=+2550.853839698" Jul 28 13:12:46.583507 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:46.583476 2573 generic.go:356] "Generic (PLEG): container finished" podID="53bb46bc-3b2f-483f-99ba-5bee6e6b564b" containerID="b2d4f482a21d1c3fb9e3f65638d36501056c9d39262bc8c33fa8682b9c110fe8" exitCode=0 Jul 28 13:12:46.583887 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:46.583532 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp" event={"ID":"53bb46bc-3b2f-483f-99ba-5bee6e6b564b","Type":"ContainerDied","Data":"b2d4f482a21d1c3fb9e3f65638d36501056c9d39262bc8c33fa8682b9c110fe8"} Jul 28 13:12:47.716051 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:47.716030 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp" Jul 28 13:12:47.738989 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:47.738965 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bb46bc-3b2f-483f-99ba-5bee6e6b564b-maas-api-service-ca\") pod \"53bb46bc-3b2f-483f-99ba-5bee6e6b564b\" (UID: \"53bb46bc-3b2f-483f-99ba-5bee6e6b564b\") " Jul 28 13:12:47.739304 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:47.739286 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bb46bc-3b2f-483f-99ba-5bee6e6b564b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53bb46bc-3b2f-483f-99ba-5bee6e6b564b" (UID: "53bb46bc-3b2f-483f-99ba-5bee6e6b564b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:12:47.840616 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:47.840521 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bb46bc-3b2f-483f-99ba-5bee6e6b564b-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:12:48.599339 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:48.599305 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp" event={"ID":"53bb46bc-3b2f-483f-99ba-5bee6e6b564b","Type":"ContainerDied","Data":"b2ff9be6d90ba73a16d633e52e0871c4c84ea41d5e7d61bef2066c69fb5ca192"} Jul 28 13:12:48.599567 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:48.599353 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2ff9be6d90ba73a16d633e52e0871c4c84ea41d5e7d61bef2066c69fb5ca192" Jul 28 13:12:48.599567 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:48.599319 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp" Jul 28 13:12:58.232541 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:58.232502 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:12:59.543915 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:59.543882 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp"] Jul 28 13:12:59.548479 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:12:59.548454 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e75cd7-vvgvp"] Jul 28 13:13:01.267662 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:01.267629 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53bb46bc-3b2f-483f-99ba-5bee6e6b564b" path="/var/lib/kubelet/pods/53bb46bc-3b2f-483f-99ba-5bee6e6b564b/volumes" Jul 28 13:13:13.231561 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:13.231524 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:13:28.928429 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:28.928393 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:13:30.929218 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:30.929180 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p"] Jul 28 13:13:30.929618 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:30.929544 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53bb46bc-3b2f-483f-99ba-5bee6e6b564b" containerName="revoke-keys" Jul 28 13:13:30.929618 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:30.929556 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="53bb46bc-3b2f-483f-99ba-5bee6e6b564b" containerName="revoke-keys" Jul 28 13:13:30.929697 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:30.929619 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="53bb46bc-3b2f-483f-99ba-5bee6e6b564b" containerName="revoke-keys" Jul 28 13:13:30.932263 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:30.932248 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p" Jul 28 13:13:30.935220 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:30.935191 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:13:30.935220 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:30.935194 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 13:13:30.939878 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:30.939851 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p"] Jul 28 13:13:31.107704 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:31.107662 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53278495-fbc7-4308-bda4-0a1e54185842-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p\" (UID: \"53278495-fbc7-4308-bda4-0a1e54185842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p" Jul 28 13:13:31.209062 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:31.208972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53278495-fbc7-4308-bda4-0a1e54185842-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p\" (UID: \"53278495-fbc7-4308-bda4-0a1e54185842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p" Jul 28 13:13:31.209687 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:31.209668 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53278495-fbc7-4308-bda4-0a1e54185842-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p\" (UID: \"53278495-fbc7-4308-bda4-0a1e54185842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p" Jul 28 13:13:31.243801 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:31.243774 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p" Jul 28 13:13:31.375846 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:31.375817 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p"] Jul 28 13:13:31.377427 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:13:31.377395 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53278495_fbc7_4308_bda4_0a1e54185842.slice/crio-27b14143430fb65a5009610ea2d11a85a9c111c2c065bd80edb363b1da1f6ee0 WatchSource:0}: Error finding container 27b14143430fb65a5009610ea2d11a85a9c111c2c065bd80edb363b1da1f6ee0: Status 404 returned error can't find the container with id 27b14143430fb65a5009610ea2d11a85a9c111c2c065bd80edb363b1da1f6ee0 Jul 28 13:13:31.831543 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:31.831513 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p" event={"ID":"53278495-fbc7-4308-bda4-0a1e54185842","Type":"ContainerStarted","Data":"39a915b2413229b86922695a73089974bdc8383d5f9e72c164c9083e685725de"} Jul 28 13:13:31.831712 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:31.831562 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p" event={"ID":"53278495-fbc7-4308-bda4-0a1e54185842","Type":"ContainerStarted","Data":"27b14143430fb65a5009610ea2d11a85a9c111c2c065bd80edb363b1da1f6ee0"} Jul 28 13:13:31.847765 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:31.847715 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p" podStartSLOduration=1.847697615 podStartE2EDuration="1.847697615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:13:31.845912955 +0000 UTC m=+2597.105203695" watchObservedRunningTime="2026-07-28 13:13:31.847697615 +0000 UTC m=+2597.106988557" Jul 28 13:13:32.838593 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:32.838553 2573 generic.go:356] "Generic (PLEG): container finished" podID="53278495-fbc7-4308-bda4-0a1e54185842" containerID="39a915b2413229b86922695a73089974bdc8383d5f9e72c164c9083e685725de" exitCode=0 Jul 28 13:13:32.838998 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:32.838651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p" event={"ID":"53278495-fbc7-4308-bda4-0a1e54185842","Type":"ContainerDied","Data":"39a915b2413229b86922695a73089974bdc8383d5f9e72c164c9083e685725de"} Jul 28 13:13:33.976737 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:33.976707 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p" Jul 28 13:13:34.032607 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:34.032569 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53278495-fbc7-4308-bda4-0a1e54185842-maas-api-service-ca\") pod \"53278495-fbc7-4308-bda4-0a1e54185842\" (UID: \"53278495-fbc7-4308-bda4-0a1e54185842\") " Jul 28 13:13:34.032944 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:34.032921 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53278495-fbc7-4308-bda4-0a1e54185842-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53278495-fbc7-4308-bda4-0a1e54185842" (UID: "53278495-fbc7-4308-bda4-0a1e54185842"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:13:34.133806 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:34.133777 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53278495-fbc7-4308-bda4-0a1e54185842-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:13:34.851587 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:34.851549 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p" event={"ID":"53278495-fbc7-4308-bda4-0a1e54185842","Type":"ContainerDied","Data":"27b14143430fb65a5009610ea2d11a85a9c111c2c065bd80edb363b1da1f6ee0"} Jul 28 13:13:34.851752 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:34.851595 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p" Jul 28 13:13:34.851752 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:34.851608 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27b14143430fb65a5009610ea2d11a85a9c111c2c065bd80edb363b1da1f6ee0" Jul 28 13:13:49.434617 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:49.434533 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:13:51.017537 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:51.017505 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p"] Jul 28 13:13:51.019359 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:51.019337 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-12b3e2-qhv9p"] Jul 28 13:13:51.267981 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:51.267905 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53278495-fbc7-4308-bda4-0a1e54185842" path="/var/lib/kubelet/pods/53278495-fbc7-4308-bda4-0a1e54185842/volumes" Jul 28 13:13:58.086876 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.086844 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t"] Jul 28 13:13:58.087309 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.087171 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53278495-fbc7-4308-bda4-0a1e54185842" containerName="revoke-keys" Jul 28 13:13:58.087309 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.087181 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="53278495-fbc7-4308-bda4-0a1e54185842" containerName="revoke-keys" Jul 28 13:13:58.087309 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.087251 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="53278495-fbc7-4308-bda4-0a1e54185842" containerName="revoke-keys" Jul 28 13:13:58.090166 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.090151 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t" Jul 28 13:13:58.096744 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.096714 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 13:13:58.096868 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.096847 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:13:58.106127 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.106106 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t"] Jul 28 13:13:58.131146 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.131116 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34dac4dd-c778-4842-9367-d4ada12188c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t\" (UID: \"34dac4dd-c778-4842-9367-d4ada12188c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t" Jul 28 13:13:58.232430 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.232405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34dac4dd-c778-4842-9367-d4ada12188c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t\" (UID: \"34dac4dd-c778-4842-9367-d4ada12188c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t" Jul 28 13:13:58.232989 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.232969 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34dac4dd-c778-4842-9367-d4ada12188c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t\" (UID: \"34dac4dd-c778-4842-9367-d4ada12188c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t" Jul 28 13:13:58.400373 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.400351 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t" Jul 28 13:13:58.532784 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.532750 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t"] Jul 28 13:13:58.534332 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:13:58.534308 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34dac4dd_c778_4842_9367_d4ada12188c3.slice/crio-cfc56b375738b1b934f7178d886fe0b5453c8e5f699523ac9475e8ddcc5c133c WatchSource:0}: Error finding container cfc56b375738b1b934f7178d886fe0b5453c8e5f699523ac9475e8ddcc5c133c: Status 404 returned error can't find the container with id cfc56b375738b1b934f7178d886fe0b5453c8e5f699523ac9475e8ddcc5c133c Jul 28 13:13:58.978865 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.978774 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t" event={"ID":"34dac4dd-c778-4842-9367-d4ada12188c3","Type":"ContainerStarted","Data":"6878f9998cc96acdb55e050c879c0b31fa2797786bdaa4076098ea36c96a49fb"} Jul 28 13:13:58.978865 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.978827 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t" event={"ID":"34dac4dd-c778-4842-9367-d4ada12188c3","Type":"ContainerStarted","Data":"cfc56b375738b1b934f7178d886fe0b5453c8e5f699523ac9475e8ddcc5c133c"} Jul 28 13:13:58.997702 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:58.997657 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t" podStartSLOduration=0.997641186 podStartE2EDuration="997.641186ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:13:58.995294145 +0000 UTC m=+2624.254584891" watchObservedRunningTime="2026-07-28 13:13:58.997641186 +0000 UTC m=+2624.256931927" Jul 28 13:13:59.986310 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:59.986273 2573 generic.go:356] "Generic (PLEG): container finished" podID="34dac4dd-c778-4842-9367-d4ada12188c3" containerID="6878f9998cc96acdb55e050c879c0b31fa2797786bdaa4076098ea36c96a49fb" exitCode=0 Jul 28 13:13:59.986712 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:13:59.986358 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t" event={"ID":"34dac4dd-c778-4842-9367-d4ada12188c3","Type":"ContainerDied","Data":"6878f9998cc96acdb55e050c879c0b31fa2797786bdaa4076098ea36c96a49fb"} Jul 28 13:14:01.130969 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:01.130947 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t" Jul 28 13:14:01.255748 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:01.255684 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34dac4dd-c778-4842-9367-d4ada12188c3-maas-api-service-ca\") pod \"34dac4dd-c778-4842-9367-d4ada12188c3\" (UID: \"34dac4dd-c778-4842-9367-d4ada12188c3\") " Jul 28 13:14:01.255998 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:01.255978 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34dac4dd-c778-4842-9367-d4ada12188c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34dac4dd-c778-4842-9367-d4ada12188c3" (UID: "34dac4dd-c778-4842-9367-d4ada12188c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:14:01.357011 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:01.356982 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34dac4dd-c778-4842-9367-d4ada12188c3-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:14:01.998308 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:01.998271 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t" event={"ID":"34dac4dd-c778-4842-9367-d4ada12188c3","Type":"ContainerDied","Data":"cfc56b375738b1b934f7178d886fe0b5453c8e5f699523ac9475e8ddcc5c133c"} Jul 28 13:14:01.998308 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:01.998294 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t" Jul 28 13:14:01.998308 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:01.998317 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfc56b375738b1b934f7178d886fe0b5453c8e5f699523ac9475e8ddcc5c133c" Jul 28 13:14:15.465284 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:15.465248 2573 scope.go:117] "RemoveContainer" containerID="a8a424122cc21c8335285eac7ae73b9ecdbcfbcf37f3c8f2b956429c1c3c7c4c" Jul 28 13:14:18.152405 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:18.152371 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t"] Jul 28 13:14:18.155276 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:18.155254 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bec5cf-tll8t"] Jul 28 13:14:18.327345 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:18.327312 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:14:19.267638 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:19.267610 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34dac4dd-c778-4842-9367-d4ada12188c3" path="/var/lib/kubelet/pods/34dac4dd-c778-4842-9367-d4ada12188c3/volumes" Jul 28 13:14:33.221270 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:33.221238 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:14:42.243955 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:42.243922 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:14:50.512743 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.512713 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9"] Jul 28 13:14:50.513115 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.513037 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34dac4dd-c778-4842-9367-d4ada12188c3" containerName="revoke-keys" Jul 28 13:14:50.513115 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.513048 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="34dac4dd-c778-4842-9367-d4ada12188c3" containerName="revoke-keys" Jul 28 13:14:50.513194 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.513133 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="34dac4dd-c778-4842-9367-d4ada12188c3" containerName="revoke-keys" Jul 28 13:14:50.515906 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.515881 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.518847 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.518828 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-67d68f\"/\"openshift-service-ca.crt\"" Jul 28 13:14:50.520448 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.520419 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-67d68f\"/\"kube-root-ca.crt\"" Jul 28 13:14:50.520533 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.520421 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-67d68f\"/\"default-dockercfg-vc2fv\"" Jul 28 13:14:50.520533 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.520423 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-67d68f\"/\"auth-test-model-67d68f-kserve-self-signed-certs\"" Jul 28 13:14:50.529776 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.529756 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9"] Jul 28 13:14:50.556972 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.556945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-dshm\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.557084 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.556979 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-model-cache\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.557084 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.557015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-tmp-dir\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.557084 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.557069 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a3d0202-6cc0-43b6-96c9-69dd790529be-tls-certs\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.557197 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.557107 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-home\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.557197 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.557126 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jbft\" (UniqueName: \"kubernetes.io/projected/3a3d0202-6cc0-43b6-96c9-69dd790529be-kube-api-access-5jbft\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.657651 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.657625 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-dshm\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.657839 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.657664 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-model-cache\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.657839 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.657723 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-tmp-dir\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.657839 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.657760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a3d0202-6cc0-43b6-96c9-69dd790529be-tls-certs\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.657839 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.657802 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-home\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.657839 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.657825 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jbft\" (UniqueName: \"kubernetes.io/projected/3a3d0202-6cc0-43b6-96c9-69dd790529be-kube-api-access-5jbft\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.658159 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.658131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-tmp-dir\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.658159 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.658146 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-model-cache\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.658266 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.658194 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-home\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.660050 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.660025 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-dshm\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.660387 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.660371 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a3d0202-6cc0-43b6-96c9-69dd790529be-tls-certs\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.666161 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.666136 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jbft\" (UniqueName: \"kubernetes.io/projected/3a3d0202-6cc0-43b6-96c9-69dd790529be-kube-api-access-5jbft\") pod \"auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.825093 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.825009 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:14:50.964021 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:50.963997 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9"] Jul 28 13:14:50.965376 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:14:50.965343 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a3d0202_6cc0_43b6_96c9_69dd790529be.slice/crio-f12687d5998015719fb7c74fe632b63531968691841e94e6a333403809f2601a WatchSource:0}: Error finding container f12687d5998015719fb7c74fe632b63531968691841e94e6a333403809f2601a: Status 404 returned error can't find the container with id f12687d5998015719fb7c74fe632b63531968691841e94e6a333403809f2601a Jul 28 13:14:51.255900 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:51.255868 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" event={"ID":"3a3d0202-6cc0-43b6-96c9-69dd790529be","Type":"ContainerStarted","Data":"c241540d29218f8008f71edca6440c41048b8c10fed1d2f5678b9e21b9688d90"} Jul 28 13:14:51.256098 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:51.255908 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" event={"ID":"3a3d0202-6cc0-43b6-96c9-69dd790529be","Type":"ContainerStarted","Data":"f12687d5998015719fb7c74fe632b63531968691841e94e6a333403809f2601a"} Jul 28 13:14:51.275889 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:51.275842 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" podStartSLOduration=1.275826457 podStartE2EDuration="1.275826457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:14:51.27346472 +0000 UTC m=+2676.532755453" watchObservedRunningTime="2026-07-28 13:14:51.275826457 +0000 UTC m=+2676.535117239" Jul 28 13:14:55.820667 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:14:55.820636 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:15:00.132026 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.131988 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754075-25t7m"] Jul 28 13:15:00.135557 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.135530 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-827464-29754075-vztfq"] Jul 28 13:15:00.135696 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.135639 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754075-25t7m" Jul 28 13:15:00.138517 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.138496 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:15:00.138632 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.138527 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-67d68f-29754075-8q99f"] Jul 28 13:15:00.138675 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.138660 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-827464-29754075-vztfq" Jul 28 13:15:00.141577 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.141559 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-67d68f-29754075-8q99f" Jul 28 13:15:00.146561 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.146546 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754075-25t7m" Jul 28 13:15:00.153883 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.153865 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-827464-29754075-vztfq" Jul 28 13:15:00.161063 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.161042 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754075-25t7m"] Jul 28 13:15:00.163101 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.163077 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-67d68f-29754075-8q99f"] Jul 28 13:15:00.166311 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.166292 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-827464-29754075-vztfq"] Jul 28 13:15:00.190492 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.190471 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-67d68f-29754075-8q99f" Jul 28 13:15:00.310544 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.310478 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754075-25t7m"] Jul 28 13:15:00.311755 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:15:00.311731 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27a355a_5dfe_4674_9e38_864f49835ef6.slice/crio-920077c11eb8d7e6a336262dd0cffebf9bdb9554fcaabddbe2d2443d5f94c39d WatchSource:0}: Error finding container 920077c11eb8d7e6a336262dd0cffebf9bdb9554fcaabddbe2d2443d5f94c39d: Status 404 returned error can't find the container with id 920077c11eb8d7e6a336262dd0cffebf9bdb9554fcaabddbe2d2443d5f94c39d Jul 28 13:15:00.341401 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.341361 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-827464-29754075-vztfq"] Jul 28 13:15:00.347958 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:15:00.347933 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode856e63d_1826_42d1_baf4_68d5723d93de.slice/crio-460c06145b8296145eb0fd73e13203bfd87ff76bf69d4f0ff79d475215375274 WatchSource:0}: Error finding container 460c06145b8296145eb0fd73e13203bfd87ff76bf69d4f0ff79d475215375274: Status 404 returned error can't find the container with id 460c06145b8296145eb0fd73e13203bfd87ff76bf69d4f0ff79d475215375274 Jul 28 13:15:00.366720 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.366696 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-67d68f-29754075-8q99f"] Jul 28 13:15:00.367427 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:15:00.367399 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad589737_8064_4393_92c7_89de51fc8703.slice/crio-01ed663d8ae19c012f97983a2fbad7b811f61fde7a0172b1a0a24e307db55fbc WatchSource:0}: Error finding container 01ed663d8ae19c012f97983a2fbad7b811f61fde7a0172b1a0a24e307db55fbc: Status 404 returned error can't find the container with id 01ed663d8ae19c012f97983a2fbad7b811f61fde7a0172b1a0a24e307db55fbc Jul 28 13:15:00.826013 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.825970 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:15:00.826221 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.826057 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:15:00.838432 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:00.838413 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:15:01.314336 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:01.314255 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754075-25t7m" event={"ID":"a27a355a-5dfe-4674-9e38-864f49835ef6","Type":"ContainerStarted","Data":"c91b7a5941fbad81bce8891ce337718585d7867667771b140d1b3de8a034aee4"} Jul 28 13:15:01.314336 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:01.314302 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754075-25t7m" event={"ID":"a27a355a-5dfe-4674-9e38-864f49835ef6","Type":"ContainerStarted","Data":"920077c11eb8d7e6a336262dd0cffebf9bdb9554fcaabddbe2d2443d5f94c39d"} Jul 28 13:15:01.316046 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:01.316024 2573 generic.go:356] "Generic (PLEG): container finished" podID="e856e63d-1826-42d1-baf4-68d5723d93de" containerID="81764b2721eb3a99d181b8979cc395ead87d6a36f5bf43c2438892c122c13c03" exitCode=0 Jul 28 13:15:01.316132 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:01.316097 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-827464-29754075-vztfq" event={"ID":"e856e63d-1826-42d1-baf4-68d5723d93de","Type":"ContainerDied","Data":"81764b2721eb3a99d181b8979cc395ead87d6a36f5bf43c2438892c122c13c03"} Jul 28 13:15:01.316181 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:01.316137 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-827464-29754075-vztfq" event={"ID":"e856e63d-1826-42d1-baf4-68d5723d93de","Type":"ContainerStarted","Data":"460c06145b8296145eb0fd73e13203bfd87ff76bf69d4f0ff79d475215375274"} Jul 28 13:15:01.317630 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:01.317609 2573 generic.go:356] "Generic (PLEG): container finished" podID="ad589737-8064-4393-92c7-89de51fc8703" containerID="57de88a49d3885f562319bcc52615a42187d9c04a5a43f2489616537947b25b4" exitCode=0 Jul 28 13:15:01.317698 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:01.317678 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-67d68f-29754075-8q99f" event={"ID":"ad589737-8064-4393-92c7-89de51fc8703","Type":"ContainerDied","Data":"57de88a49d3885f562319bcc52615a42187d9c04a5a43f2489616537947b25b4"} Jul 28 13:15:01.317738 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:01.317717 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-67d68f-29754075-8q99f" event={"ID":"ad589737-8064-4393-92c7-89de51fc8703","Type":"ContainerStarted","Data":"01ed663d8ae19c012f97983a2fbad7b811f61fde7a0172b1a0a24e307db55fbc"} Jul 28 13:15:01.328894 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:01.328874 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:15:01.332940 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:01.332904 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754075-25t7m" podStartSLOduration=1.332892781 podStartE2EDuration="1.332892781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:15:01.331085813 +0000 UTC m=+2686.590376553" watchObservedRunningTime="2026-07-28 13:15:01.332892781 +0000 UTC m=+2686.592183563" Jul 28 13:15:02.325219 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:02.325187 2573 generic.go:356] "Generic (PLEG): container finished" podID="a27a355a-5dfe-4674-9e38-864f49835ef6" containerID="c91b7a5941fbad81bce8891ce337718585d7867667771b140d1b3de8a034aee4" exitCode=0 Jul 28 13:15:02.325711 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:02.325245 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754075-25t7m" event={"ID":"a27a355a-5dfe-4674-9e38-864f49835ef6","Type":"ContainerDied","Data":"c91b7a5941fbad81bce8891ce337718585d7867667771b140d1b3de8a034aee4"} Jul 28 13:15:02.476909 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:02.476883 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-827464-29754075-vztfq" Jul 28 13:15:02.503162 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:02.503140 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-67d68f-29754075-8q99f" Jul 28 13:15:03.331956 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:03.331930 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-67d68f-29754075-8q99f" Jul 28 13:15:03.332327 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:03.331971 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-67d68f-29754075-8q99f" event={"ID":"ad589737-8064-4393-92c7-89de51fc8703","Type":"ContainerDied","Data":"01ed663d8ae19c012f97983a2fbad7b811f61fde7a0172b1a0a24e307db55fbc"} Jul 28 13:15:03.332327 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:03.332017 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01ed663d8ae19c012f97983a2fbad7b811f61fde7a0172b1a0a24e307db55fbc" Jul 28 13:15:03.333451 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:03.333418 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-827464-29754075-vztfq" event={"ID":"e856e63d-1826-42d1-baf4-68d5723d93de","Type":"ContainerDied","Data":"460c06145b8296145eb0fd73e13203bfd87ff76bf69d4f0ff79d475215375274"} Jul 28 13:15:03.333555 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:03.333456 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-827464-29754075-vztfq" Jul 28 13:15:03.333555 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:03.333488 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="460c06145b8296145eb0fd73e13203bfd87ff76bf69d4f0ff79d475215375274" Jul 28 13:15:03.469400 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:03.469361 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754075-25t7m" Jul 28 13:15:04.339666 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:04.339635 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754075-25t7m" Jul 28 13:15:04.340031 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:04.339671 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754075-25t7m" event={"ID":"a27a355a-5dfe-4674-9e38-864f49835ef6","Type":"ContainerDied","Data":"920077c11eb8d7e6a336262dd0cffebf9bdb9554fcaabddbe2d2443d5f94c39d"} Jul 28 13:15:04.340031 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:04.339712 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="920077c11eb8d7e6a336262dd0cffebf9bdb9554fcaabddbe2d2443d5f94c39d" Jul 28 13:15:15.125004 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:15.124971 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:15:15.486185 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:15.486146 2573 scope.go:117] "RemoveContainer" containerID="634474b09a39323e12790768566516a9b2368b2f0f8edd26c6cc7088555e4ff8" Jul 28 13:15:41.926766 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:15:41.926733 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:16:02.931215 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:16:02.931179 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:16:21.447530 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:16:21.447493 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:16:43.029738 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:16:43.029702 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:17:01.841410 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:17:01.841351 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:17:24.533328 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:17:24.533290 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:17:42.925168 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:17:42.925136 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:18:09.929148 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:18:09.929064 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:18:23.639790 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:18:23.639754 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:18:34.535509 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:18:34.535473 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:18:53.240399 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:18:53.240368 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:19:12.327013 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:12.326963 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:19:15.536244 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:15.536212 2573 scope.go:117] "RemoveContainer" containerID="b2d4f482a21d1c3fb9e3f65638d36501056c9d39262bc8c33fa8682b9c110fe8" Jul 28 13:19:15.546412 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:15.546386 2573 scope.go:117] "RemoveContainer" containerID="410e0c6ac8d6060b73919f03ff3347322a58fd09c06689a57b66832d870d2856" Jul 28 13:19:35.242154 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:35.242119 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:19:39.431955 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:39.431878 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9"] Jul 28 13:19:51.237272 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:51.237238 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:19:54.451319 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.451276 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" podUID="3a3d0202-6cc0-43b6-96c9-69dd790529be" containerName="main" containerID="cri-o://c241540d29218f8008f71edca6440c41048b8c10fed1d2f5678b9e21b9688d90" gracePeriod=45 Jul 28 13:19:54.700180 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.700159 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:19:54.813586 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.813506 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jbft\" (UniqueName: \"kubernetes.io/projected/3a3d0202-6cc0-43b6-96c9-69dd790529be-kube-api-access-5jbft\") pod \"3a3d0202-6cc0-43b6-96c9-69dd790529be\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " Jul 28 13:19:54.813586 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.813538 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-model-cache\") pod \"3a3d0202-6cc0-43b6-96c9-69dd790529be\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " Jul 28 13:19:54.813586 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.813553 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-home\") pod \"3a3d0202-6cc0-43b6-96c9-69dd790529be\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " Jul 28 13:19:54.813866 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.813622 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-tmp-dir\") pod \"3a3d0202-6cc0-43b6-96c9-69dd790529be\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " Jul 28 13:19:54.813866 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.813676 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a3d0202-6cc0-43b6-96c9-69dd790529be-tls-certs\") pod \"3a3d0202-6cc0-43b6-96c9-69dd790529be\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " Jul 28 13:19:54.813866 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.813743 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-dshm\") pod \"3a3d0202-6cc0-43b6-96c9-69dd790529be\" (UID: \"3a3d0202-6cc0-43b6-96c9-69dd790529be\") " Jul 28 13:19:54.814030 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.813931 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a3d0202-6cc0-43b6-96c9-69dd790529be" (UID: "3a3d0202-6cc0-43b6-96c9-69dd790529be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:19:54.814030 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.813947 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-home" (OuterVolumeSpecName: "home") pod "3a3d0202-6cc0-43b6-96c9-69dd790529be" (UID: "3a3d0202-6cc0-43b6-96c9-69dd790529be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:19:54.814170 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.813838 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a3d0202-6cc0-43b6-96c9-69dd790529be" (UID: "3a3d0202-6cc0-43b6-96c9-69dd790529be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:19:54.816017 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.815987 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a3d0202-6cc0-43b6-96c9-69dd790529be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a3d0202-6cc0-43b6-96c9-69dd790529be" (UID: "3a3d0202-6cc0-43b6-96c9-69dd790529be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:19:54.816107 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.816019 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3d0202-6cc0-43b6-96c9-69dd790529be-kube-api-access-5jbft" (OuterVolumeSpecName: "kube-api-access-5jbft") pod "3a3d0202-6cc0-43b6-96c9-69dd790529be" (UID: "3a3d0202-6cc0-43b6-96c9-69dd790529be"). InnerVolumeSpecName "kube-api-access-5jbft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:19:54.816159 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.816092 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-dshm" (OuterVolumeSpecName: "dshm") pod "3a3d0202-6cc0-43b6-96c9-69dd790529be" (UID: "3a3d0202-6cc0-43b6-96c9-69dd790529be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:19:54.892219 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.892190 2573 generic.go:356] "Generic (PLEG): container finished" podID="3a3d0202-6cc0-43b6-96c9-69dd790529be" containerID="c241540d29218f8008f71edca6440c41048b8c10fed1d2f5678b9e21b9688d90" exitCode=0 Jul 28 13:19:54.892395 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.892233 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" Jul 28 13:19:54.892395 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.892290 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" event={"ID":"3a3d0202-6cc0-43b6-96c9-69dd790529be","Type":"ContainerDied","Data":"c241540d29218f8008f71edca6440c41048b8c10fed1d2f5678b9e21b9688d90"} Jul 28 13:19:54.892395 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.892328 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9" event={"ID":"3a3d0202-6cc0-43b6-96c9-69dd790529be","Type":"ContainerDied","Data":"f12687d5998015719fb7c74fe632b63531968691841e94e6a333403809f2601a"} Jul 28 13:19:54.892395 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.892344 2573 scope.go:117] "RemoveContainer" containerID="c241540d29218f8008f71edca6440c41048b8c10fed1d2f5678b9e21b9688d90" Jul 28 13:19:54.902102 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.902084 2573 scope.go:117] "RemoveContainer" containerID="c241540d29218f8008f71edca6440c41048b8c10fed1d2f5678b9e21b9688d90" Jul 28 13:19:54.902369 ip-10-0-136-145 kubenswrapper[2573]: E0728 13:19:54.902350 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c241540d29218f8008f71edca6440c41048b8c10fed1d2f5678b9e21b9688d90\": container with ID starting with c241540d29218f8008f71edca6440c41048b8c10fed1d2f5678b9e21b9688d90 not found: ID does not exist" containerID="c241540d29218f8008f71edca6440c41048b8c10fed1d2f5678b9e21b9688d90" Jul 28 13:19:54.902421 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.902377 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c241540d29218f8008f71edca6440c41048b8c10fed1d2f5678b9e21b9688d90"} err="failed to get container status \"c241540d29218f8008f71edca6440c41048b8c10fed1d2f5678b9e21b9688d90\": rpc error: code = NotFound desc = could not find container \"c241540d29218f8008f71edca6440c41048b8c10fed1d2f5678b9e21b9688d90\": container with ID starting with c241540d29218f8008f71edca6440c41048b8c10fed1d2f5678b9e21b9688d90 not found: ID does not exist" Jul 28 13:19:54.914783 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.914599 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9"] Jul 28 13:19:54.915467 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.915427 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-dshm\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:19:54.915609 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.915478 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jbft\" (UniqueName: \"kubernetes.io/projected/3a3d0202-6cc0-43b6-96c9-69dd790529be-kube-api-access-5jbft\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:19:54.915609 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.915493 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-model-cache\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:19:54.915609 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.915508 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-home\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:19:54.915609 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.915543 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3d0202-6cc0-43b6-96c9-69dd790529be-tmp-dir\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:19:54.915609 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.915561 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a3d0202-6cc0-43b6-96c9-69dd790529be-tls-certs\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:19:54.922449 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:54.922416 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-67d68f/auth-test-model-67d68f-kserve-ccc47d7cf-cbfr9"] Jul 28 13:19:55.268472 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:19:55.268420 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a3d0202-6cc0-43b6-96c9-69dd790529be" path="/var/lib/kubelet/pods/3a3d0202-6cc0-43b6-96c9-69dd790529be/volumes" Jul 28 13:20:07.923815 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:07.923775 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:20:15.570953 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:15.570918 2573 scope.go:117] "RemoveContainer" containerID="6878f9998cc96acdb55e050c879c0b31fa2797786bdaa4076098ea36c96a49fb" Jul 28 13:20:15.578930 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:15.578902 2573 scope.go:117] "RemoveContainer" containerID="39a915b2413229b86922695a73089974bdc8383d5f9e72c164c9083e685725de" Jul 28 13:20:28.435003 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.434971 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f"] Jul 28 13:20:28.435391 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.435310 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad589737-8064-4393-92c7-89de51fc8703" containerName="cleanup" Jul 28 13:20:28.435391 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.435320 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad589737-8064-4393-92c7-89de51fc8703" containerName="cleanup" Jul 28 13:20:28.435391 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.435333 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e856e63d-1826-42d1-baf4-68d5723d93de" containerName="cleanup" Jul 28 13:20:28.435391 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.435338 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e856e63d-1826-42d1-baf4-68d5723d93de" containerName="cleanup" Jul 28 13:20:28.435391 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.435346 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a3d0202-6cc0-43b6-96c9-69dd790529be" containerName="main" Jul 28 13:20:28.435391 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.435352 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a3d0202-6cc0-43b6-96c9-69dd790529be" containerName="main" Jul 28 13:20:28.435391 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.435364 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27a355a-5dfe-4674-9e38-864f49835ef6" containerName="cleanup" Jul 28 13:20:28.435391 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.435369 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27a355a-5dfe-4674-9e38-864f49835ef6" containerName="cleanup" Jul 28 13:20:28.435744 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.435418 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e856e63d-1826-42d1-baf4-68d5723d93de" containerName="cleanup" Jul 28 13:20:28.435744 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.435426 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a3d0202-6cc0-43b6-96c9-69dd790529be" containerName="main" Jul 28 13:20:28.435744 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.435433 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27a355a-5dfe-4674-9e38-864f49835ef6" containerName="cleanup" Jul 28 13:20:28.435744 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.435457 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad589737-8064-4393-92c7-89de51fc8703" containerName="cleanup" Jul 28 13:20:28.438264 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.438249 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.442884 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.442863 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-827464\"/\"openshift-service-ca.crt\"" Jul 28 13:20:28.443025 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.442906 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-827464\"/\"default-dockercfg-hgq5j\"" Jul 28 13:20:28.443025 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.442913 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-827464\"/\"kube-root-ca.crt\"" Jul 28 13:20:28.443025 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.442978 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-827464\"/\"sub-test-model-827464-kserve-self-signed-certs\"" Jul 28 13:20:28.449925 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.449901 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f"] Jul 28 13:20:28.516549 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.516522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqh8\" (UniqueName: \"kubernetes.io/projected/6a990b30-cfbb-4a5d-93e0-3c878db296ad-kube-api-access-zlqh8\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.516673 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.516554 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-model-cache\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.516673 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.516593 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-home\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.516673 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.516634 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-dshm\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.516781 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.516675 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-tmp-dir\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.516781 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.516705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a990b30-cfbb-4a5d-93e0-3c878db296ad-tls-certs\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.618110 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.618069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqh8\" (UniqueName: \"kubernetes.io/projected/6a990b30-cfbb-4a5d-93e0-3c878db296ad-kube-api-access-zlqh8\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.618110 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.618110 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-model-cache\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.618361 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.618150 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-home\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.618361 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.618173 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-dshm\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.618361 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.618207 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-tmp-dir\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.618361 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.618244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a990b30-cfbb-4a5d-93e0-3c878db296ad-tls-certs\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.618674 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.618651 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-home\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.618735 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.618714 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-model-cache\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.618798 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.618749 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-tmp-dir\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.620782 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.620760 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-dshm\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.621040 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.621023 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a990b30-cfbb-4a5d-93e0-3c878db296ad-tls-certs\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.628369 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.628335 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqh8\" (UniqueName: \"kubernetes.io/projected/6a990b30-cfbb-4a5d-93e0-3c878db296ad-kube-api-access-zlqh8\") pod \"sub-test-model-827464-kserve-66d566d985-xbd6f\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.747622 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.747534 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:28.889382 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:28.889345 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f"] Jul 28 13:20:28.891672 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:20:28.891619 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a990b30_cfbb_4a5d_93e0_3c878db296ad.slice/crio-aefa7ee46ac6a6f7985ce9da06a4f571ed076e8ea1c2e549d3aa760796606ef0 WatchSource:0}: Error finding container aefa7ee46ac6a6f7985ce9da06a4f571ed076e8ea1c2e549d3aa760796606ef0: Status 404 returned error can't find the container with id aefa7ee46ac6a6f7985ce9da06a4f571ed076e8ea1c2e549d3aa760796606ef0 Jul 28 13:20:29.073008 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:29.072914 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" event={"ID":"6a990b30-cfbb-4a5d-93e0-3c878db296ad","Type":"ContainerStarted","Data":"bad04e26087ffde7fd852df3cf8972d110b4d80ff5700bf49b8502314ff944e6"} Jul 28 13:20:29.073008 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:29.072956 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" event={"ID":"6a990b30-cfbb-4a5d-93e0-3c878db296ad","Type":"ContainerStarted","Data":"aefa7ee46ac6a6f7985ce9da06a4f571ed076e8ea1c2e549d3aa760796606ef0"} Jul 28 13:20:29.090789 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:29.090739 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" podStartSLOduration=1.090722095 podStartE2EDuration="1.090722095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:20:29.089623425 +0000 UTC m=+3014.348914167" watchObservedRunningTime="2026-07-28 13:20:29.090722095 +0000 UTC m=+3014.350012836" Jul 28 13:20:32.223354 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:32.223315 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:20:38.748344 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:38.748303 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:38.748344 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:38.748351 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:38.761041 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:38.761016 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:20:39.136356 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:20:39.136330 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:21:04.443316 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:04.443279 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:21:23.775910 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:23.775832 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:21:29.044641 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:29.044608 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f"] Jul 28 13:21:40.327744 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:40.327705 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:21:44.064512 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.064428 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" podUID="6a990b30-cfbb-4a5d-93e0-3c878db296ad" containerName="main" containerID="cri-o://bad04e26087ffde7fd852df3cf8972d110b4d80ff5700bf49b8502314ff944e6" gracePeriod=45 Jul 28 13:21:44.323014 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.322956 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:21:44.381609 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.381571 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-dshm\") pod \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " Jul 28 13:21:44.381801 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.381621 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a990b30-cfbb-4a5d-93e0-3c878db296ad-tls-certs\") pod \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " Jul 28 13:21:44.381801 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.381643 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-model-cache\") pod \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " Jul 28 13:21:44.381801 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.381674 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlqh8\" (UniqueName: \"kubernetes.io/projected/6a990b30-cfbb-4a5d-93e0-3c878db296ad-kube-api-access-zlqh8\") pod \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " Jul 28 13:21:44.381972 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.381833 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-home\") pod \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " Jul 28 13:21:44.381972 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.381873 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-tmp-dir\") pod \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\" (UID: \"6a990b30-cfbb-4a5d-93e0-3c878db296ad\") " Jul 28 13:21:44.382073 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.381975 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a990b30-cfbb-4a5d-93e0-3c878db296ad" (UID: "6a990b30-cfbb-4a5d-93e0-3c878db296ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:21:44.382167 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.382144 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-home" (OuterVolumeSpecName: "home") pod "6a990b30-cfbb-4a5d-93e0-3c878db296ad" (UID: "6a990b30-cfbb-4a5d-93e0-3c878db296ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:21:44.382167 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.382154 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a990b30-cfbb-4a5d-93e0-3c878db296ad" (UID: "6a990b30-cfbb-4a5d-93e0-3c878db296ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:21:44.382262 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.382233 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-model-cache\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:21:44.382262 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.382252 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-home\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:21:44.382369 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.382317 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-tmp-dir\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:21:44.384036 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.384012 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-dshm" (OuterVolumeSpecName: "dshm") pod "6a990b30-cfbb-4a5d-93e0-3c878db296ad" (UID: "6a990b30-cfbb-4a5d-93e0-3c878db296ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:21:44.384134 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.384112 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a990b30-cfbb-4a5d-93e0-3c878db296ad-kube-api-access-zlqh8" (OuterVolumeSpecName: "kube-api-access-zlqh8") pod "6a990b30-cfbb-4a5d-93e0-3c878db296ad" (UID: "6a990b30-cfbb-4a5d-93e0-3c878db296ad"). InnerVolumeSpecName "kube-api-access-zlqh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:21:44.384207 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.384191 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a990b30-cfbb-4a5d-93e0-3c878db296ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a990b30-cfbb-4a5d-93e0-3c878db296ad" (UID: "6a990b30-cfbb-4a5d-93e0-3c878db296ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:21:44.472799 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.472759 2573 generic.go:356] "Generic (PLEG): container finished" podID="6a990b30-cfbb-4a5d-93e0-3c878db296ad" containerID="bad04e26087ffde7fd852df3cf8972d110b4d80ff5700bf49b8502314ff944e6" exitCode=0 Jul 28 13:21:44.472962 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.472807 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" Jul 28 13:21:44.472962 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.472807 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" event={"ID":"6a990b30-cfbb-4a5d-93e0-3c878db296ad","Type":"ContainerDied","Data":"bad04e26087ffde7fd852df3cf8972d110b4d80ff5700bf49b8502314ff944e6"} Jul 28 13:21:44.472962 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.472846 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f" event={"ID":"6a990b30-cfbb-4a5d-93e0-3c878db296ad","Type":"ContainerDied","Data":"aefa7ee46ac6a6f7985ce9da06a4f571ed076e8ea1c2e549d3aa760796606ef0"} Jul 28 13:21:44.472962 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.472869 2573 scope.go:117] "RemoveContainer" containerID="bad04e26087ffde7fd852df3cf8972d110b4d80ff5700bf49b8502314ff944e6" Jul 28 13:21:44.482316 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.482295 2573 scope.go:117] "RemoveContainer" containerID="bad04e26087ffde7fd852df3cf8972d110b4d80ff5700bf49b8502314ff944e6" Jul 28 13:21:44.482607 ip-10-0-136-145 kubenswrapper[2573]: E0728 13:21:44.482581 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad04e26087ffde7fd852df3cf8972d110b4d80ff5700bf49b8502314ff944e6\": container with ID starting with bad04e26087ffde7fd852df3cf8972d110b4d80ff5700bf49b8502314ff944e6 not found: ID does not exist" containerID="bad04e26087ffde7fd852df3cf8972d110b4d80ff5700bf49b8502314ff944e6" Jul 28 13:21:44.482674 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.482620 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad04e26087ffde7fd852df3cf8972d110b4d80ff5700bf49b8502314ff944e6"} err="failed to get container status \"bad04e26087ffde7fd852df3cf8972d110b4d80ff5700bf49b8502314ff944e6\": rpc error: code = NotFound desc = could not find container \"bad04e26087ffde7fd852df3cf8972d110b4d80ff5700bf49b8502314ff944e6\": container with ID starting with bad04e26087ffde7fd852df3cf8972d110b4d80ff5700bf49b8502314ff944e6 not found: ID does not exist" Jul 28 13:21:44.482806 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.482793 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a990b30-cfbb-4a5d-93e0-3c878db296ad-dshm\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:21:44.482850 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.482810 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a990b30-cfbb-4a5d-93e0-3c878db296ad-tls-certs\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:21:44.482850 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.482819 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlqh8\" (UniqueName: \"kubernetes.io/projected/6a990b30-cfbb-4a5d-93e0-3c878db296ad-kube-api-access-zlqh8\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:21:44.500218 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.500187 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f"] Jul 28 13:21:44.502161 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:44.502139 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-827464/sub-test-model-827464-kserve-66d566d985-xbd6f"] Jul 28 13:21:45.267370 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:45.267337 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a990b30-cfbb-4a5d-93e0-3c878db296ad" path="/var/lib/kubelet/pods/6a990b30-cfbb-4a5d-93e0-3c878db296ad/volumes" Jul 28 13:21:59.244345 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.244313 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx"] Jul 28 13:21:59.244793 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.244695 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a990b30-cfbb-4a5d-93e0-3c878db296ad" containerName="main" Jul 28 13:21:59.244793 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.244707 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a990b30-cfbb-4a5d-93e0-3c878db296ad" containerName="main" Jul 28 13:21:59.244793 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.244774 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a990b30-cfbb-4a5d-93e0-3c878db296ad" containerName="main" Jul 28 13:21:59.247760 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.247746 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.250787 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.250768 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-827464\"/\"kube-root-ca.crt\"" Jul 28 13:21:59.252406 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.252387 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-827464\"/\"openshift-service-ca.crt\"" Jul 28 13:21:59.252559 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.252534 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-827464\"/\"rate-test-model-827464-kserve-self-signed-certs\"" Jul 28 13:21:59.252682 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.252586 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-827464\"/\"default-dockercfg-hgq5j\"" Jul 28 13:21:59.258028 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.258005 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx"] Jul 28 13:21:59.308943 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.308909 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-tls-certs\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.309120 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.308952 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-home\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.309120 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.309012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-dshm\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.309120 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.309050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-tmp-dir\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.309120 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.309104 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-model-cache\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.309273 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.309158 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9j6p\" (UniqueName: \"kubernetes.io/projected/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-kube-api-access-q9j6p\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.410126 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.410091 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-model-cache\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.410310 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.410145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9j6p\" (UniqueName: \"kubernetes.io/projected/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-kube-api-access-q9j6p\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.410310 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.410189 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-tls-certs\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.410310 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.410213 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-home\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.410310 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.410234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-dshm\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.410310 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.410257 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-tmp-dir\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.410606 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.410580 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-model-cache\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.410666 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.410627 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-tmp-dir\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.410727 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.410703 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-home\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.412616 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.412592 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-dshm\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.413006 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.412985 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-tls-certs\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.419871 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.419852 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9j6p\" (UniqueName: \"kubernetes.io/projected/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-kube-api-access-q9j6p\") pod \"rate-test-model-827464-kserve-975c6bdc9-r2djx\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.557767 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.557669 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:21:59.700332 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:21:59.697943 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx"] Jul 28 13:22:00.558411 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:22:00.558369 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" event={"ID":"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9","Type":"ContainerStarted","Data":"c3d8a0a5b4ab04b8d6a38425ca0a8e6c54ee46c2187ef08f69cd7c4d180e94c2"} Jul 28 13:22:00.558411 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:22:00.558417 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" event={"ID":"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9","Type":"ContainerStarted","Data":"fe1d71992a7d6876bf94f0a08837c43fa49eafe301107c0d24bb9ad5e41948b3"} Jul 28 13:22:00.579629 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:22:00.579574 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" podStartSLOduration=1.579558892 podStartE2EDuration="1.579558892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:22:00.577613593 +0000 UTC m=+3105.836904337" watchObservedRunningTime="2026-07-28 13:22:00.579558892 +0000 UTC m=+3105.838849633" Jul 28 13:22:03.028193 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:22:03.028155 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:22:09.558791 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:22:09.558758 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:22:09.559254 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:22:09.558810 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:22:09.571382 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:22:09.571347 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:22:09.616872 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:22:09.616848 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:22:58.331964 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:22:58.331925 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:23:13.326151 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:13.326115 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:23:27.639139 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:27.639103 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:23:32.718653 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:32.718617 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx"] Jul 28 13:23:36.628585 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:36.628550 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7487-140018767229872"] Jul 28 13:23:36.631982 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:36.631963 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-140018767229872" Jul 28 13:23:36.634740 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:36.634717 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zcgkh\"" Jul 28 13:23:36.639642 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:36.639619 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-140018767229872"] Jul 28 13:23:36.657875 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:36.657840 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n62gv\" (UniqueName: \"kubernetes.io/projected/889c0f98-f0fe-4f35-8496-9c93de05828e-kube-api-access-n62gv\") pod \"test-curl-7487-140018767229872\" (UID: \"889c0f98-f0fe-4f35-8496-9c93de05828e\") " pod="opendatahub/test-curl-7487-140018767229872" Jul 28 13:23:36.758971 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:36.758935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n62gv\" (UniqueName: \"kubernetes.io/projected/889c0f98-f0fe-4f35-8496-9c93de05828e-kube-api-access-n62gv\") pod \"test-curl-7487-140018767229872\" (UID: \"889c0f98-f0fe-4f35-8496-9c93de05828e\") " pod="opendatahub/test-curl-7487-140018767229872" Jul 28 13:23:36.772954 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:36.772932 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n62gv\" (UniqueName: \"kubernetes.io/projected/889c0f98-f0fe-4f35-8496-9c93de05828e-kube-api-access-n62gv\") pod \"test-curl-7487-140018767229872\" (UID: \"889c0f98-f0fe-4f35-8496-9c93de05828e\") " pod="opendatahub/test-curl-7487-140018767229872" Jul 28 13:23:36.942212 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:36.942179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-140018767229872" Jul 28 13:23:37.086234 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:37.086197 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-140018767229872"] Jul 28 13:23:37.086561 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:23:37.086539 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod889c0f98_f0fe_4f35_8496_9c93de05828e.slice/crio-3423e7f3070767d38d10e74bf317c65b62fccf5e79364dd09ff9f70cb9be4ec7 WatchSource:0}: Error finding container 3423e7f3070767d38d10e74bf317c65b62fccf5e79364dd09ff9f70cb9be4ec7: Status 404 returned error can't find the container with id 3423e7f3070767d38d10e74bf317c65b62fccf5e79364dd09ff9f70cb9be4ec7 Jul 28 13:23:37.087823 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:37.087807 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 13:23:38.073051 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:38.072966 2573 generic.go:356] "Generic (PLEG): container finished" podID="889c0f98-f0fe-4f35-8496-9c93de05828e" containerID="8d307eb7cbee708dd962edaa81fe26ac8243d6aad78b6ccf53ebe9f273f7e5aa" exitCode=0 Jul 28 13:23:38.073051 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:38.073037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-140018767229872" event={"ID":"889c0f98-f0fe-4f35-8496-9c93de05828e","Type":"ContainerDied","Data":"8d307eb7cbee708dd962edaa81fe26ac8243d6aad78b6ccf53ebe9f273f7e5aa"} Jul 28 13:23:38.073453 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:38.073064 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-140018767229872" event={"ID":"889c0f98-f0fe-4f35-8496-9c93de05828e","Type":"ContainerStarted","Data":"3423e7f3070767d38d10e74bf317c65b62fccf5e79364dd09ff9f70cb9be4ec7"} Jul 28 13:23:39.199710 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.199682 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-140018767229872" Jul 28 13:23:39.233508 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.233471 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7487-140018767229872_889c0f98-f0fe-4f35-8496-9c93de05828e/test-curl-7487-140018767229872/0.log" Jul 28 13:23:39.259531 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.259491 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7487-140018767229872"] Jul 28 13:23:39.270107 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.270083 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7487-140018767229872"] Jul 28 13:23:39.281510 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.281487 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n62gv\" (UniqueName: \"kubernetes.io/projected/889c0f98-f0fe-4f35-8496-9c93de05828e-kube-api-access-n62gv\") pod \"889c0f98-f0fe-4f35-8496-9c93de05828e\" (UID: \"889c0f98-f0fe-4f35-8496-9c93de05828e\") " Jul 28 13:23:39.283821 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.283795 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889c0f98-f0fe-4f35-8496-9c93de05828e-kube-api-access-n62gv" (OuterVolumeSpecName: "kube-api-access-n62gv") pod "889c0f98-f0fe-4f35-8496-9c93de05828e" (UID: "889c0f98-f0fe-4f35-8496-9c93de05828e"). InnerVolumeSpecName "kube-api-access-n62gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:23:39.382524 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.382389 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n62gv\" (UniqueName: \"kubernetes.io/projected/889c0f98-f0fe-4f35-8496-9c93de05828e-kube-api-access-n62gv\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:23:39.431501 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.431461 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7487-140018767128592"] Jul 28 13:23:39.431900 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.431881 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="889c0f98-f0fe-4f35-8496-9c93de05828e" containerName="test-curl-7487-140018767229872" Jul 28 13:23:39.431993 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.431904 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="889c0f98-f0fe-4f35-8496-9c93de05828e" containerName="test-curl-7487-140018767229872" Jul 28 13:23:39.431993 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.431987 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="889c0f98-f0fe-4f35-8496-9c93de05828e" containerName="test-curl-7487-140018767229872" Jul 28 13:23:39.435031 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.435011 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-140018767128592" Jul 28 13:23:39.442266 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.442071 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-140018767128592"] Jul 28 13:23:39.483367 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.483333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvkj4\" (UniqueName: \"kubernetes.io/projected/8dc52693-8f4e-4fb7-8f37-e5d850ca6568-kube-api-access-lvkj4\") pod \"test-curl-7487-140018767128592\" (UID: \"8dc52693-8f4e-4fb7-8f37-e5d850ca6568\") " pod="opendatahub/test-curl-7487-140018767128592" Jul 28 13:23:39.584020 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.583980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvkj4\" (UniqueName: \"kubernetes.io/projected/8dc52693-8f4e-4fb7-8f37-e5d850ca6568-kube-api-access-lvkj4\") pod \"test-curl-7487-140018767128592\" (UID: \"8dc52693-8f4e-4fb7-8f37-e5d850ca6568\") " pod="opendatahub/test-curl-7487-140018767128592" Jul 28 13:23:39.592523 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.592479 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvkj4\" (UniqueName: \"kubernetes.io/projected/8dc52693-8f4e-4fb7-8f37-e5d850ca6568-kube-api-access-lvkj4\") pod \"test-curl-7487-140018767128592\" (UID: \"8dc52693-8f4e-4fb7-8f37-e5d850ca6568\") " pod="opendatahub/test-curl-7487-140018767128592" Jul 28 13:23:39.745241 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.745195 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-140018767128592" Jul 28 13:23:39.877424 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:39.877397 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-140018767128592"] Jul 28 13:23:39.878649 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:23:39.878621 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dc52693_8f4e_4fb7_8f37_e5d850ca6568.slice/crio-a460253db0053b6a43ab75032246861a8fdcb14e09709d4e01b7d39d2ab74abd WatchSource:0}: Error finding container a460253db0053b6a43ab75032246861a8fdcb14e09709d4e01b7d39d2ab74abd: Status 404 returned error can't find the container with id a460253db0053b6a43ab75032246861a8fdcb14e09709d4e01b7d39d2ab74abd Jul 28 13:23:40.086420 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:40.086335 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-140018767229872" Jul 28 13:23:40.086420 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:40.086357 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3423e7f3070767d38d10e74bf317c65b62fccf5e79364dd09ff9f70cb9be4ec7" Jul 28 13:23:40.087757 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:40.087724 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-140018767128592" event={"ID":"8dc52693-8f4e-4fb7-8f37-e5d850ca6568","Type":"ContainerStarted","Data":"a460253db0053b6a43ab75032246861a8fdcb14e09709d4e01b7d39d2ab74abd"} Jul 28 13:23:40.098006 ip-10-0-136-145 kubenswrapper[2573]: E0728 13:23:40.097977 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7487-140018767229872\" is forbidden: User \"system:node:ip-10-0-136-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-145.ec2.internal' and this object" podUID="889c0f98-f0fe-4f35-8496-9c93de05828e" pod="opendatahub/test-curl-7487-140018767229872" Jul 28 13:23:41.094865 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:41.094832 2573 generic.go:356] "Generic (PLEG): container finished" podID="8dc52693-8f4e-4fb7-8f37-e5d850ca6568" containerID="f263f3063b42c020fedaa3f94ccd05e5bbcc86e2e0c959ff8ddca21bb0e7bf77" exitCode=0 Jul 28 13:23:41.095251 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:41.094913 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-140018767128592" event={"ID":"8dc52693-8f4e-4fb7-8f37-e5d850ca6568","Type":"ContainerDied","Data":"f263f3063b42c020fedaa3f94ccd05e5bbcc86e2e0c959ff8ddca21bb0e7bf77"} Jul 28 13:23:41.111469 ip-10-0-136-145 kubenswrapper[2573]: E0728 13:23:41.111420 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7487-140018767229872\" is forbidden: User \"system:node:ip-10-0-136-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-145.ec2.internal' and this object" podUID="889c0f98-f0fe-4f35-8496-9c93de05828e" pod="opendatahub/test-curl-7487-140018767229872" Jul 28 13:23:41.267756 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:41.267725 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="889c0f98-f0fe-4f35-8496-9c93de05828e" path="/var/lib/kubelet/pods/889c0f98-f0fe-4f35-8496-9c93de05828e/volumes" Jul 28 13:23:42.228321 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:42.228298 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-140018767128592" Jul 28 13:23:42.249040 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:42.249012 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7487-140018767128592_8dc52693-8f4e-4fb7-8f37-e5d850ca6568/test-curl-7487-140018767128592/0.log" Jul 28 13:23:42.274774 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:42.274746 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7487-140018767128592"] Jul 28 13:23:42.276829 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:42.276810 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7487-140018767128592"] Jul 28 13:23:42.309002 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:42.308973 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvkj4\" (UniqueName: \"kubernetes.io/projected/8dc52693-8f4e-4fb7-8f37-e5d850ca6568-kube-api-access-lvkj4\") pod \"8dc52693-8f4e-4fb7-8f37-e5d850ca6568\" (UID: \"8dc52693-8f4e-4fb7-8f37-e5d850ca6568\") " Jul 28 13:23:42.311317 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:42.311295 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc52693-8f4e-4fb7-8f37-e5d850ca6568-kube-api-access-lvkj4" (OuterVolumeSpecName: "kube-api-access-lvkj4") pod "8dc52693-8f4e-4fb7-8f37-e5d850ca6568" (UID: "8dc52693-8f4e-4fb7-8f37-e5d850ca6568"). InnerVolumeSpecName "kube-api-access-lvkj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:23:42.409767 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:42.409734 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvkj4\" (UniqueName: \"kubernetes.io/projected/8dc52693-8f4e-4fb7-8f37-e5d850ca6568-kube-api-access-lvkj4\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:23:43.106780 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:43.106750 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-140018767128592" Jul 28 13:23:43.106951 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:43.106794 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a460253db0053b6a43ab75032246861a8fdcb14e09709d4e01b7d39d2ab74abd" Jul 28 13:23:43.117623 ip-10-0-136-145 kubenswrapper[2573]: E0728 13:23:43.117597 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7487-140018767128592\" is forbidden: User \"system:node:ip-10-0-136-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-145.ec2.internal' and this object" podUID="8dc52693-8f4e-4fb7-8f37-e5d850ca6568" pod="opendatahub/test-curl-7487-140018767128592" Jul 28 13:23:43.268302 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:43.268272 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dc52693-8f4e-4fb7-8f37-e5d850ca6568" path="/var/lib/kubelet/pods/8dc52693-8f4e-4fb7-8f37-e5d850ca6568/volumes" Jul 28 13:23:43.819847 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:43.819813 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:23:47.737674 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:47.737611 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" podUID="d874e5f1-0b19-4e6d-bab9-4832aa7d64e9" containerName="main" containerID="cri-o://c3d8a0a5b4ab04b8d6a38425ca0a8e6c54ee46c2187ef08f69cd7c4d180e94c2" gracePeriod=45 Jul 28 13:23:47.988509 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:47.988451 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:23:48.060125 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.060086 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-tls-certs\") pod \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " Jul 28 13:23:48.060315 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.060145 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-model-cache\") pod \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " Jul 28 13:23:48.060315 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.060184 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-dshm\") pod \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " Jul 28 13:23:48.060315 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.060217 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9j6p\" (UniqueName: \"kubernetes.io/projected/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-kube-api-access-q9j6p\") pod \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " Jul 28 13:23:48.060315 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.060242 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-home\") pod \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " Jul 28 13:23:48.060315 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.060268 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-tmp-dir\") pod \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\" (UID: \"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9\") " Jul 28 13:23:48.060612 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.060492 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "d874e5f1-0b19-4e6d-bab9-4832aa7d64e9" (UID: "d874e5f1-0b19-4e6d-bab9-4832aa7d64e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:23:48.060612 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.060535 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-home" (OuterVolumeSpecName: "home") pod "d874e5f1-0b19-4e6d-bab9-4832aa7d64e9" (UID: "d874e5f1-0b19-4e6d-bab9-4832aa7d64e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:23:48.060758 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.060732 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d874e5f1-0b19-4e6d-bab9-4832aa7d64e9" (UID: "d874e5f1-0b19-4e6d-bab9-4832aa7d64e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:23:48.062554 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.062528 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-kube-api-access-q9j6p" (OuterVolumeSpecName: "kube-api-access-q9j6p") pod "d874e5f1-0b19-4e6d-bab9-4832aa7d64e9" (UID: "d874e5f1-0b19-4e6d-bab9-4832aa7d64e9"). InnerVolumeSpecName "kube-api-access-q9j6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:23:48.062658 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.062573 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-dshm" (OuterVolumeSpecName: "dshm") pod "d874e5f1-0b19-4e6d-bab9-4832aa7d64e9" (UID: "d874e5f1-0b19-4e6d-bab9-4832aa7d64e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:23:48.062658 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.062579 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d874e5f1-0b19-4e6d-bab9-4832aa7d64e9" (UID: "d874e5f1-0b19-4e6d-bab9-4832aa7d64e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:23:48.134654 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.134616 2573 generic.go:356] "Generic (PLEG): container finished" podID="d874e5f1-0b19-4e6d-bab9-4832aa7d64e9" containerID="c3d8a0a5b4ab04b8d6a38425ca0a8e6c54ee46c2187ef08f69cd7c4d180e94c2" exitCode=0 Jul 28 13:23:48.134654 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.134659 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" event={"ID":"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9","Type":"ContainerDied","Data":"c3d8a0a5b4ab04b8d6a38425ca0a8e6c54ee46c2187ef08f69cd7c4d180e94c2"} Jul 28 13:23:48.134870 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.134661 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" Jul 28 13:23:48.134870 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.134687 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx" event={"ID":"d874e5f1-0b19-4e6d-bab9-4832aa7d64e9","Type":"ContainerDied","Data":"fe1d71992a7d6876bf94f0a08837c43fa49eafe301107c0d24bb9ad5e41948b3"} Jul 28 13:23:48.134870 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.134707 2573 scope.go:117] "RemoveContainer" containerID="c3d8a0a5b4ab04b8d6a38425ca0a8e6c54ee46c2187ef08f69cd7c4d180e94c2" Jul 28 13:23:48.144749 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.144727 2573 scope.go:117] "RemoveContainer" containerID="c3d8a0a5b4ab04b8d6a38425ca0a8e6c54ee46c2187ef08f69cd7c4d180e94c2" Jul 28 13:23:48.145022 ip-10-0-136-145 kubenswrapper[2573]: E0728 13:23:48.145004 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d8a0a5b4ab04b8d6a38425ca0a8e6c54ee46c2187ef08f69cd7c4d180e94c2\": container with ID starting with c3d8a0a5b4ab04b8d6a38425ca0a8e6c54ee46c2187ef08f69cd7c4d180e94c2 not found: ID does not exist" containerID="c3d8a0a5b4ab04b8d6a38425ca0a8e6c54ee46c2187ef08f69cd7c4d180e94c2" Jul 28 13:23:48.145078 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.145031 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d8a0a5b4ab04b8d6a38425ca0a8e6c54ee46c2187ef08f69cd7c4d180e94c2"} err="failed to get container status \"c3d8a0a5b4ab04b8d6a38425ca0a8e6c54ee46c2187ef08f69cd7c4d180e94c2\": rpc error: code = NotFound desc = could not find container \"c3d8a0a5b4ab04b8d6a38425ca0a8e6c54ee46c2187ef08f69cd7c4d180e94c2\": container with ID starting with c3d8a0a5b4ab04b8d6a38425ca0a8e6c54ee46c2187ef08f69cd7c4d180e94c2 not found: ID does not exist" Jul 28 13:23:48.161640 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.161621 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-tls-certs\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:23:48.161640 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.161640 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-model-cache\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:23:48.161779 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.161648 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-dshm\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:23:48.161779 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.161657 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9j6p\" (UniqueName: \"kubernetes.io/projected/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-kube-api-access-q9j6p\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:23:48.161779 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.161665 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-home\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:23:48.161779 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.161690 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9-tmp-dir\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:23:48.172919 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.172895 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx"] Jul 28 13:23:48.181825 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:48.181803 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-827464/rate-test-model-827464-kserve-975c6bdc9-r2djx"] Jul 28 13:23:49.267895 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:23:49.267849 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d874e5f1-0b19-4e6d-bab9-4832aa7d64e9" path="/var/lib/kubelet/pods/d874e5f1-0b19-4e6d-bab9-4832aa7d64e9/volumes" Jul 28 13:24:03.421147 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:03.421113 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:24:13.402677 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.402601 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz"] Jul 28 13:24:13.403110 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.402930 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dc52693-8f4e-4fb7-8f37-e5d850ca6568" containerName="test-curl-7487-140018767128592" Jul 28 13:24:13.403110 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.402941 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dc52693-8f4e-4fb7-8f37-e5d850ca6568" containerName="test-curl-7487-140018767128592" Jul 28 13:24:13.403110 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.402952 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d874e5f1-0b19-4e6d-bab9-4832aa7d64e9" containerName="main" Jul 28 13:24:13.403110 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.402957 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d874e5f1-0b19-4e6d-bab9-4832aa7d64e9" containerName="main" Jul 28 13:24:13.403110 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.403018 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d874e5f1-0b19-4e6d-bab9-4832aa7d64e9" containerName="main" Jul 28 13:24:13.403110 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.403026 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dc52693-8f4e-4fb7-8f37-e5d850ca6568" containerName="test-curl-7487-140018767128592" Jul 28 13:24:13.406555 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.406536 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:13.409878 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.409853 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-06a4d2-serving-cert\"" Jul 28 13:24:13.410023 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.409883 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 13:24:13.411302 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.411284 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d58lq\"" Jul 28 13:24:13.411412 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.411327 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 13:24:13.417641 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.417621 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz"] Jul 28 13:24:13.481681 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.481626 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd84bd6-b977-427e-9c1e-acf2da21a36d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz\" (UID: \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:13.481681 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.481683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzcdj\" (UniqueName: \"kubernetes.io/projected/5fd84bd6-b977-427e-9c1e-acf2da21a36d-kube-api-access-kzcdj\") pod \"maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz\" (UID: \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:13.481923 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.481762 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd84bd6-b977-427e-9c1e-acf2da21a36d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz\" (UID: \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:13.582679 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.582648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd84bd6-b977-427e-9c1e-acf2da21a36d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz\" (UID: \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:13.582679 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.582691 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzcdj\" (UniqueName: \"kubernetes.io/projected/5fd84bd6-b977-427e-9c1e-acf2da21a36d-kube-api-access-kzcdj\") pod \"maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz\" (UID: \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:13.582950 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.582750 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd84bd6-b977-427e-9c1e-acf2da21a36d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz\" (UID: \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:13.582950 ip-10-0-136-145 kubenswrapper[2573]: E0728 13:24:13.582915 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-serving-cert: secret "maas-api-e2e-ipp-b-06a4d2-serving-cert" not found Jul 28 13:24:13.583070 ip-10-0-136-145 kubenswrapper[2573]: E0728 13:24:13.582987 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fd84bd6-b977-427e-9c1e-acf2da21a36d-maas-api-tls podName:5fd84bd6-b977-427e-9c1e-acf2da21a36d nodeName:}" failed. No retries permitted until 2026-07-28 13:24:14.08296424 +0000 UTC m=+3239.342254962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5fd84bd6-b977-427e-9c1e-acf2da21a36d-maas-api-tls") pod "maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" (UID: "5fd84bd6-b977-427e-9c1e-acf2da21a36d") : secret "maas-api-e2e-ipp-b-06a4d2-serving-cert" not found Jul 28 13:24:13.583301 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.583277 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd84bd6-b977-427e-9c1e-acf2da21a36d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz\" (UID: \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:13.595147 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:13.595121 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzcdj\" (UniqueName: \"kubernetes.io/projected/5fd84bd6-b977-427e-9c1e-acf2da21a36d-kube-api-access-kzcdj\") pod \"maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz\" (UID: \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:14.088396 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:14.088359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd84bd6-b977-427e-9c1e-acf2da21a36d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz\" (UID: \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:14.091191 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:14.091166 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd84bd6-b977-427e-9c1e-acf2da21a36d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz\" (UID: \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:14.318686 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:14.318652 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:14.460325 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:14.460298 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz"] Jul 28 13:24:14.462682 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:24:14.462652 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd84bd6_b977_427e_9c1e_acf2da21a36d.slice/crio-a896e3b0d3aba651d305219dc9f367a5f70c904a51f6e9ff5748a6c630bfe931 WatchSource:0}: Error finding container a896e3b0d3aba651d305219dc9f367a5f70c904a51f6e9ff5748a6c630bfe931: Status 404 returned error can't find the container with id a896e3b0d3aba651d305219dc9f367a5f70c904a51f6e9ff5748a6c630bfe931 Jul 28 13:24:15.295263 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:15.295229 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" event={"ID":"5fd84bd6-b977-427e-9c1e-acf2da21a36d","Type":"ContainerStarted","Data":"a896e3b0d3aba651d305219dc9f367a5f70c904a51f6e9ff5748a6c630bfe931"} Jul 28 13:24:17.309419 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:17.309368 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" event={"ID":"5fd84bd6-b977-427e-9c1e-acf2da21a36d","Type":"ContainerStarted","Data":"7d295cca4f051acac0238a01a261e77b0dfe0f9e73970530739de542304c0fe1"} Jul 28 13:24:17.309921 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:17.309503 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:17.328290 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:17.328243 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" podStartSLOduration=1.911466664 podStartE2EDuration="4.32822631s" totalImagesPullingTime="2.416759646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:24:17.326942308 +0000 UTC m=+3242.586233052" watchObservedRunningTime="2026-07-28 13:24:17.32822631 +0000 UTC m=+3242.587517050" Jul 28 13:24:22.329244 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:22.329199 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:24:23.320604 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:23.320565 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:24:38.221885 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.221846 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x"] Jul 28 13:24:38.226159 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.226141 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.230700 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.230675 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8fb56b\"/\"kube-root-ca.crt\"" Jul 28 13:24:38.230837 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.230677 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8fb56b\"/\"default-dockercfg-w7dxr\"" Jul 28 13:24:38.230837 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.230678 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8fb56b\"/\"ipp-route-8fb56b-kserve-self-signed-certs\"" Jul 28 13:24:38.230960 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.230681 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8fb56b\"/\"openshift-service-ca.crt\"" Jul 28 13:24:38.237117 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.237089 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x"] Jul 28 13:24:38.309551 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.309521 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-home\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.309748 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.309627 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-dshm\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.309748 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.309663 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-model-cache\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.309885 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.309754 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-tls-certs\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.309885 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.309867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm44w\" (UniqueName: \"kubernetes.io/projected/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-kube-api-access-mm44w\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.309991 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.309914 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-tmp-dir\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.411290 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.411246 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-dshm\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.411290 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.411300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-model-cache\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.411570 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.411361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-tls-certs\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.411570 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.411431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm44w\" (UniqueName: \"kubernetes.io/projected/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-kube-api-access-mm44w\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.411570 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.411485 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-tmp-dir\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.411570 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.411508 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-home\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.411852 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.411837 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-home\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.411913 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.411888 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-model-cache\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.412409 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.412379 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-tmp-dir\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.414402 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.414381 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-tls-certs\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.414516 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.414408 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-dshm\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.421949 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.421926 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm44w\" (UniqueName: \"kubernetes.io/projected/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-kube-api-access-mm44w\") pod \"ipp-route-8fb56b-kserve-79c9dcb747-wgr8x\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.537292 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.537197 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:38.680900 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:38.680873 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x"] Jul 28 13:24:38.682387 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:24:38.682351 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07e2500f_d2d1_4234_ab6e_1c5b726d2ce5.slice/crio-6c46da304ef810366955dc3e1641d7908548b085f1ba48e2be7f0582d7c753ad WatchSource:0}: Error finding container 6c46da304ef810366955dc3e1641d7908548b085f1ba48e2be7f0582d7c753ad: Status 404 returned error can't find the container with id 6c46da304ef810366955dc3e1641d7908548b085f1ba48e2be7f0582d7c753ad Jul 28 13:24:39.426500 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:39.426468 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" event={"ID":"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5","Type":"ContainerStarted","Data":"99a240523893cf9e9547a5aecacd0410d4b3015425d6be65912351426120170c"} Jul 28 13:24:39.426907 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:39.426511 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" event={"ID":"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5","Type":"ContainerStarted","Data":"6c46da304ef810366955dc3e1641d7908548b085f1ba48e2be7f0582d7c753ad"} Jul 28 13:24:39.444082 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:39.444035 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" podStartSLOduration=1.44401976 podStartE2EDuration="1.44401976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:24:39.441958218 +0000 UTC m=+3264.701248958" watchObservedRunningTime="2026-07-28 13:24:39.44401976 +0000 UTC m=+3264.703310528" Jul 28 13:24:42.026431 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:42.026385 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:24:48.538218 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:48.538186 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:48.538744 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:48.538235 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:48.550522 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:48.550497 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:24:49.491914 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:24:49.491882 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:25:10.127295 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:10.127259 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:25:25.842370 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:25.842336 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z"] Jul 28 13:25:25.844909 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:25.844890 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:25.847779 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:25.847750 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-33eb65-serving-cert\"" Jul 28 13:25:25.857590 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:25.857568 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z"] Jul 28 13:25:25.944811 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:25.944779 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rbk\" (UniqueName: \"kubernetes.io/projected/94d1ae24-319d-4908-ab8e-955b3bb78b91-kube-api-access-59rbk\") pod \"maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z\" (UID: \"94d1ae24-319d-4908-ab8e-955b3bb78b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:25.944811 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:25.944829 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d1ae24-319d-4908-ab8e-955b3bb78b91-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z\" (UID: \"94d1ae24-319d-4908-ab8e-955b3bb78b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:25.945103 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:25.944965 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d1ae24-319d-4908-ab8e-955b3bb78b91-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z\" (UID: \"94d1ae24-319d-4908-ab8e-955b3bb78b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:26.045757 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:26.045718 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d1ae24-319d-4908-ab8e-955b3bb78b91-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z\" (UID: \"94d1ae24-319d-4908-ab8e-955b3bb78b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:26.045944 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:26.045772 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rbk\" (UniqueName: \"kubernetes.io/projected/94d1ae24-319d-4908-ab8e-955b3bb78b91-kube-api-access-59rbk\") pod \"maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z\" (UID: \"94d1ae24-319d-4908-ab8e-955b3bb78b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:26.045944 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:26.045799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d1ae24-319d-4908-ab8e-955b3bb78b91-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z\" (UID: \"94d1ae24-319d-4908-ab8e-955b3bb78b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:26.046431 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:26.046408 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d1ae24-319d-4908-ab8e-955b3bb78b91-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z\" (UID: \"94d1ae24-319d-4908-ab8e-955b3bb78b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:26.048395 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:26.048374 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d1ae24-319d-4908-ab8e-955b3bb78b91-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z\" (UID: \"94d1ae24-319d-4908-ab8e-955b3bb78b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:26.055567 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:26.055543 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rbk\" (UniqueName: \"kubernetes.io/projected/94d1ae24-319d-4908-ab8e-955b3bb78b91-kube-api-access-59rbk\") pod \"maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z\" (UID: \"94d1ae24-319d-4908-ab8e-955b3bb78b91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:26.155795 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:26.155761 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:26.300973 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:26.300948 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z"] Jul 28 13:25:26.303127 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:25:26.303101 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d1ae24_319d_4908_ab8e_955b3bb78b91.slice/crio-9bc3a6090582a486007ec0ba5f8fe76b2f329dc386314f55478b0415f310d5ec WatchSource:0}: Error finding container 9bc3a6090582a486007ec0ba5f8fe76b2f329dc386314f55478b0415f310d5ec: Status 404 returned error can't find the container with id 9bc3a6090582a486007ec0ba5f8fe76b2f329dc386314f55478b0415f310d5ec Jul 28 13:25:26.678071 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:26.678015 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" event={"ID":"94d1ae24-319d-4908-ab8e-955b3bb78b91","Type":"ContainerStarted","Data":"9bc3a6090582a486007ec0ba5f8fe76b2f329dc386314f55478b0415f310d5ec"} Jul 28 13:25:27.685068 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:27.685030 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" event={"ID":"94d1ae24-319d-4908-ab8e-955b3bb78b91","Type":"ContainerStarted","Data":"f13c2a0591079c7924705ba258e7571e31918a78415d5299607b34131f094abb"} Jul 28 13:25:27.685548 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:27.685094 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:27.706142 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:27.706086 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" podStartSLOduration=2.350643557 podStartE2EDuration="2.706070003s" totalImagesPullingTime="355.426446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:25:27.703677293 +0000 UTC m=+3312.962968036" watchObservedRunningTime="2026-07-28 13:25:27.706070003 +0000 UTC m=+3312.965360744" Jul 28 13:25:33.696583 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:33.696551 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:35.527103 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:35.527063 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:25:41.160713 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:41.160630 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px"] Jul 28 13:25:41.163121 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:41.163105 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px" Jul 28 13:25:41.166050 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:41.166029 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:25:41.172752 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:41.172725 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px"] Jul 28 13:25:41.288029 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:41.287989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px\" (UID: \"2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px" Jul 28 13:25:41.389224 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:41.389187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px\" (UID: \"2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px" Jul 28 13:25:41.389879 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:41.389859 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px\" (UID: \"2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px" Jul 28 13:25:41.475029 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:41.474932 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px" Jul 28 13:25:41.611265 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:41.611235 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px"] Jul 28 13:25:41.612529 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:25:41.612499 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba2ea06_8b0f_4ff1_b43a_0e677e59eab3.slice/crio-c6744a19b820b9e8d4d5bb7acaa73b71ed2f13cb9be59f6f3ab4c872101a5219 WatchSource:0}: Error finding container c6744a19b820b9e8d4d5bb7acaa73b71ed2f13cb9be59f6f3ab4c872101a5219: Status 404 returned error can't find the container with id c6744a19b820b9e8d4d5bb7acaa73b71ed2f13cb9be59f6f3ab4c872101a5219 Jul 28 13:25:41.762104 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:41.762014 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px" event={"ID":"2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3","Type":"ContainerStarted","Data":"c6744a19b820b9e8d4d5bb7acaa73b71ed2f13cb9be59f6f3ab4c872101a5219"} Jul 28 13:25:42.769297 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:42.769261 2573 generic.go:356] "Generic (PLEG): container finished" podID="2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3" containerID="a3d93f16fdcb109a3fe059865395d817786a7dd93aed462b5c42b12496f886d6" exitCode=0 Jul 28 13:25:42.769718 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:42.769322 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px" event={"ID":"2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3","Type":"ContainerDied","Data":"a3d93f16fdcb109a3fe059865395d817786a7dd93aed462b5c42b12496f886d6"} Jul 28 13:25:43.907934 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:43.907899 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px" Jul 28 13:25:44.013684 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:44.013652 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3-maas-api-service-ca\") pod \"2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3\" (UID: \"2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3\") " Jul 28 13:25:44.013989 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:44.013967 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3" (UID: "2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:25:44.114999 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:44.114908 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:25:44.781371 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:44.781333 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px" event={"ID":"2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3","Type":"ContainerDied","Data":"c6744a19b820b9e8d4d5bb7acaa73b71ed2f13cb9be59f6f3ab4c872101a5219"} Jul 28 13:25:44.781589 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:44.781401 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6744a19b820b9e8d4d5bb7acaa73b71ed2f13cb9be59f6f3ab4c872101a5219" Jul 28 13:25:44.781589 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:44.781349 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px" Jul 28 13:25:51.548022 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.547991 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z"] Jul 28 13:25:51.548423 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.548215 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" podUID="94d1ae24-319d-4908-ab8e-955b3bb78b91" containerName="maas-api" containerID="cri-o://f13c2a0591079c7924705ba258e7571e31918a78415d5299607b34131f094abb" gracePeriod=30 Jul 28 13:25:51.807571 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.807515 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:51.820289 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.820259 2573 generic.go:356] "Generic (PLEG): container finished" podID="94d1ae24-319d-4908-ab8e-955b3bb78b91" containerID="f13c2a0591079c7924705ba258e7571e31918a78415d5299607b34131f094abb" exitCode=0 Jul 28 13:25:51.820471 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.820298 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" Jul 28 13:25:51.820471 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.820341 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" event={"ID":"94d1ae24-319d-4908-ab8e-955b3bb78b91","Type":"ContainerDied","Data":"f13c2a0591079c7924705ba258e7571e31918a78415d5299607b34131f094abb"} Jul 28 13:25:51.820471 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.820403 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z" event={"ID":"94d1ae24-319d-4908-ab8e-955b3bb78b91","Type":"ContainerDied","Data":"9bc3a6090582a486007ec0ba5f8fe76b2f329dc386314f55478b0415f310d5ec"} Jul 28 13:25:51.820471 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.820456 2573 scope.go:117] "RemoveContainer" containerID="f13c2a0591079c7924705ba258e7571e31918a78415d5299607b34131f094abb" Jul 28 13:25:51.831622 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.831596 2573 scope.go:117] "RemoveContainer" containerID="f13c2a0591079c7924705ba258e7571e31918a78415d5299607b34131f094abb" Jul 28 13:25:51.831927 ip-10-0-136-145 kubenswrapper[2573]: E0728 13:25:51.831905 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13c2a0591079c7924705ba258e7571e31918a78415d5299607b34131f094abb\": container with ID starting with f13c2a0591079c7924705ba258e7571e31918a78415d5299607b34131f094abb not found: ID does not exist" containerID="f13c2a0591079c7924705ba258e7571e31918a78415d5299607b34131f094abb" Jul 28 13:25:51.832009 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.831934 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13c2a0591079c7924705ba258e7571e31918a78415d5299607b34131f094abb"} err="failed to get container status \"f13c2a0591079c7924705ba258e7571e31918a78415d5299607b34131f094abb\": rpc error: code = NotFound desc = could not find container \"f13c2a0591079c7924705ba258e7571e31918a78415d5299607b34131f094abb\": container with ID starting with f13c2a0591079c7924705ba258e7571e31918a78415d5299607b34131f094abb not found: ID does not exist" Jul 28 13:25:51.885524 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.885495 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59rbk\" (UniqueName: \"kubernetes.io/projected/94d1ae24-319d-4908-ab8e-955b3bb78b91-kube-api-access-59rbk\") pod \"94d1ae24-319d-4908-ab8e-955b3bb78b91\" (UID: \"94d1ae24-319d-4908-ab8e-955b3bb78b91\") " Jul 28 13:25:51.885697 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.885547 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d1ae24-319d-4908-ab8e-955b3bb78b91-maas-api-tls\") pod \"94d1ae24-319d-4908-ab8e-955b3bb78b91\" (UID: \"94d1ae24-319d-4908-ab8e-955b3bb78b91\") " Jul 28 13:25:51.885697 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.885590 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d1ae24-319d-4908-ab8e-955b3bb78b91-openshift-service-ca\") pod \"94d1ae24-319d-4908-ab8e-955b3bb78b91\" (UID: \"94d1ae24-319d-4908-ab8e-955b3bb78b91\") " Jul 28 13:25:51.885958 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.885934 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d1ae24-319d-4908-ab8e-955b3bb78b91-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94d1ae24-319d-4908-ab8e-955b3bb78b91" (UID: "94d1ae24-319d-4908-ab8e-955b3bb78b91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:25:51.887977 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.887951 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d1ae24-319d-4908-ab8e-955b3bb78b91-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94d1ae24-319d-4908-ab8e-955b3bb78b91" (UID: "94d1ae24-319d-4908-ab8e-955b3bb78b91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:25:51.887977 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.887954 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d1ae24-319d-4908-ab8e-955b3bb78b91-kube-api-access-59rbk" (OuterVolumeSpecName: "kube-api-access-59rbk") pod "94d1ae24-319d-4908-ab8e-955b3bb78b91" (UID: "94d1ae24-319d-4908-ab8e-955b3bb78b91"). InnerVolumeSpecName "kube-api-access-59rbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:25:51.986396 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.986359 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d1ae24-319d-4908-ab8e-955b3bb78b91-openshift-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:25:51.986396 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.986392 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59rbk\" (UniqueName: \"kubernetes.io/projected/94d1ae24-319d-4908-ab8e-955b3bb78b91-kube-api-access-59rbk\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:25:51.986396 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:51.986407 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d1ae24-319d-4908-ab8e-955b3bb78b91-maas-api-tls\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:25:52.143451 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:52.143399 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z"] Jul 28 13:25:52.150467 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:52.149245 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-33eb65-6bdd58bcf6-zqx2z"] Jul 28 13:25:53.268290 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:53.268255 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94d1ae24-319d-4908-ab8e-955b3bb78b91" path="/var/lib/kubelet/pods/94d1ae24-319d-4908-ab8e-955b3bb78b91/volumes" Jul 28 13:25:56.244644 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:56.244588 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px"] Jul 28 13:25:56.246720 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:56.246699 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-33eb65-h94px"] Jul 28 13:25:57.268173 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:57.268145 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3" path="/var/lib/kubelet/pods/2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3/volumes" Jul 28 13:25:59.029841 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:25:59.029803 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:26:07.161552 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.161501 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz"] Jul 28 13:26:07.161947 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.161880 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94d1ae24-319d-4908-ab8e-955b3bb78b91" containerName="maas-api" Jul 28 13:26:07.161947 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.161892 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="94d1ae24-319d-4908-ab8e-955b3bb78b91" containerName="maas-api" Jul 28 13:26:07.161947 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.161919 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3" containerName="revoke-keys" Jul 28 13:26:07.161947 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.161925 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3" containerName="revoke-keys" Jul 28 13:26:07.162082 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.161982 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="94d1ae24-319d-4908-ab8e-955b3bb78b91" containerName="maas-api" Jul 28 13:26:07.162082 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.161990 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba2ea06-8b0f-4ff1-b43a-0e677e59eab3" containerName="revoke-keys" Jul 28 13:26:07.164077 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.164061 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz" Jul 28 13:26:07.167015 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.166997 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:26:07.173196 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.173163 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz"] Jul 28 13:26:07.224335 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.224296 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4095f807-6f5b-4dc5-9f56-ddb7e303ea81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz\" (UID: \"4095f807-6f5b-4dc5-9f56-ddb7e303ea81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz" Jul 28 13:26:07.325617 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.325579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4095f807-6f5b-4dc5-9f56-ddb7e303ea81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz\" (UID: \"4095f807-6f5b-4dc5-9f56-ddb7e303ea81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz" Jul 28 13:26:07.326189 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.326173 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4095f807-6f5b-4dc5-9f56-ddb7e303ea81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz\" (UID: \"4095f807-6f5b-4dc5-9f56-ddb7e303ea81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz" Jul 28 13:26:07.475865 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.475760 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz" Jul 28 13:26:07.611025 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.611000 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz"] Jul 28 13:26:07.613205 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:26:07.613176 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4095f807_6f5b_4dc5_9f56_ddb7e303ea81.slice/crio-817e16dc2abe21eb5e8f290d4d16979284bac79649d98d17edd9057c400b2906 WatchSource:0}: Error finding container 817e16dc2abe21eb5e8f290d4d16979284bac79649d98d17edd9057c400b2906: Status 404 returned error can't find the container with id 817e16dc2abe21eb5e8f290d4d16979284bac79649d98d17edd9057c400b2906 Jul 28 13:26:07.913169 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.913130 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz" event={"ID":"4095f807-6f5b-4dc5-9f56-ddb7e303ea81","Type":"ContainerStarted","Data":"55ea4acf0ad07545137728a506f93a8b64b1488c425dae3c409274453d0a3915"} Jul 28 13:26:07.913341 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.913196 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz" event={"ID":"4095f807-6f5b-4dc5-9f56-ddb7e303ea81","Type":"ContainerStarted","Data":"817e16dc2abe21eb5e8f290d4d16979284bac79649d98d17edd9057c400b2906"} Jul 28 13:26:07.928246 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:07.928194 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz" podStartSLOduration=0.928178128 podStartE2EDuration="928.178128ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:26:07.927609894 +0000 UTC m=+3353.186900660" watchObservedRunningTime="2026-07-28 13:26:07.928178128 +0000 UTC m=+3353.187468868" Jul 28 13:26:08.920541 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:08.920504 2573 generic.go:356] "Generic (PLEG): container finished" podID="4095f807-6f5b-4dc5-9f56-ddb7e303ea81" containerID="55ea4acf0ad07545137728a506f93a8b64b1488c425dae3c409274453d0a3915" exitCode=0 Jul 28 13:26:08.920921 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:08.920559 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz" event={"ID":"4095f807-6f5b-4dc5-9f56-ddb7e303ea81","Type":"ContainerDied","Data":"55ea4acf0ad07545137728a506f93a8b64b1488c425dae3c409274453d0a3915"} Jul 28 13:26:10.046051 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:10.046027 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz" Jul 28 13:26:10.151513 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:10.151483 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4095f807-6f5b-4dc5-9f56-ddb7e303ea81-maas-api-service-ca\") pod \"4095f807-6f5b-4dc5-9f56-ddb7e303ea81\" (UID: \"4095f807-6f5b-4dc5-9f56-ddb7e303ea81\") " Jul 28 13:26:10.151836 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:10.151813 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4095f807-6f5b-4dc5-9f56-ddb7e303ea81-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4095f807-6f5b-4dc5-9f56-ddb7e303ea81" (UID: "4095f807-6f5b-4dc5-9f56-ddb7e303ea81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:26:10.252713 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:10.252623 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4095f807-6f5b-4dc5-9f56-ddb7e303ea81-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:26:10.932357 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:10.932313 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz" event={"ID":"4095f807-6f5b-4dc5-9f56-ddb7e303ea81","Type":"ContainerDied","Data":"817e16dc2abe21eb5e8f290d4d16979284bac79649d98d17edd9057c400b2906"} Jul 28 13:26:10.932357 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:10.932333 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz" Jul 28 13:26:10.932594 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:10.932394 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="817e16dc2abe21eb5e8f290d4d16979284bac79649d98d17edd9057c400b2906" Jul 28 13:26:20.096801 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:20.096768 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vdv2d"] Jul 28 13:26:20.097214 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:20.097120 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4095f807-6f5b-4dc5-9f56-ddb7e303ea81" containerName="revoke-keys" Jul 28 13:26:20.097214 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:20.097131 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4095f807-6f5b-4dc5-9f56-ddb7e303ea81" containerName="revoke-keys" Jul 28 13:26:20.097214 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:20.097193 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4095f807-6f5b-4dc5-9f56-ddb7e303ea81" containerName="revoke-keys" Jul 28 13:26:20.099653 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:20.099637 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" Jul 28 13:26:20.102656 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:20.102634 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-8gr7b\"" Jul 28 13:26:20.108590 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:20.108567 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vdv2d"] Jul 28 13:26:20.135880 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:20.135851 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dfxb\" (UniqueName: \"kubernetes.io/projected/9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab-kube-api-access-7dfxb\") pod \"kuadrant-operator-catalog-vdv2d\" (UID: \"9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab\") " pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" Jul 28 13:26:20.237389 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:20.237348 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dfxb\" (UniqueName: \"kubernetes.io/projected/9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab-kube-api-access-7dfxb\") pod \"kuadrant-operator-catalog-vdv2d\" (UID: \"9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab\") " pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" Jul 28 13:26:20.245850 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:20.245821 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dfxb\" (UniqueName: \"kubernetes.io/projected/9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab-kube-api-access-7dfxb\") pod \"kuadrant-operator-catalog-vdv2d\" (UID: \"9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab\") " pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" Jul 28 13:26:20.409916 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:20.409882 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" Jul 28 13:26:20.548744 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:20.548714 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vdv2d"] Jul 28 13:26:20.550053 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:26:20.550025 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5c8c83_7ccd_4d2b_b48c_8ff6ad3540ab.slice/crio-415a1ea308bbe8df8022e3a34674e3f3188835208b3ca57028d408fa96fb6b31 WatchSource:0}: Error finding container 415a1ea308bbe8df8022e3a34674e3f3188835208b3ca57028d408fa96fb6b31: Status 404 returned error can't find the container with id 415a1ea308bbe8df8022e3a34674e3f3188835208b3ca57028d408fa96fb6b31 Jul 28 13:26:20.988935 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:20.988894 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" event={"ID":"9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab","Type":"ContainerStarted","Data":"415a1ea308bbe8df8022e3a34674e3f3188835208b3ca57028d408fa96fb6b31"} Jul 28 13:26:23.002625 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:23.002594 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" event={"ID":"9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab","Type":"ContainerStarted","Data":"f820719f646fe39259122c77de1b4085207d1630df8d334bba6cf317f10bb0b6"} Jul 28 13:26:23.019809 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:23.019755 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" podStartSLOduration=0.956405165 podStartE2EDuration="3.019739438s" totalImagesPullingTime="2.063334273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:26:23.018855996 +0000 UTC m=+3368.278146748" watchObservedRunningTime="2026-07-28 13:26:23.019739438 +0000 UTC m=+3368.279030176" Jul 28 13:26:23.728059 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:23.728024 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:26:27.232603 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:27.232571 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz"] Jul 28 13:26:27.234789 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:27.234752 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8fb56b-bcsfz"] Jul 28 13:26:27.267637 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:27.267605 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4095f807-6f5b-4dc5-9f56-ddb7e303ea81" path="/var/lib/kubelet/pods/4095f807-6f5b-4dc5-9f56-ddb7e303ea81/volumes" Jul 28 13:26:30.411085 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:30.411045 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" Jul 28 13:26:30.411633 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:30.411122 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" Jul 28 13:26:30.433743 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:30.433704 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" Jul 28 13:26:31.069008 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:31.068978 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" Jul 28 13:26:31.104906 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:31.104869 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vdv2d"] Jul 28 13:26:32.605606 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:32.605570 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x"] Jul 28 13:26:33.057604 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:33.057551 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" podUID="9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab" containerName="registry-server" containerID="cri-o://f820719f646fe39259122c77de1b4085207d1630df8d334bba6cf317f10bb0b6" gracePeriod=2 Jul 28 13:26:33.299293 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:33.299271 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" Jul 28 13:26:33.356155 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:33.356068 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dfxb\" (UniqueName: \"kubernetes.io/projected/9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab-kube-api-access-7dfxb\") pod \"9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab\" (UID: \"9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab\") " Jul 28 13:26:33.358297 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:33.358276 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab-kube-api-access-7dfxb" (OuterVolumeSpecName: "kube-api-access-7dfxb") pod "9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab" (UID: "9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab"). InnerVolumeSpecName "kube-api-access-7dfxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:26:33.457668 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:33.457635 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dfxb\" (UniqueName: \"kubernetes.io/projected/9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab-kube-api-access-7dfxb\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:26:34.023485 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:34.023452 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:26:34.064251 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:34.064219 2573 generic.go:356] "Generic (PLEG): container finished" podID="9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab" containerID="f820719f646fe39259122c77de1b4085207d1630df8d334bba6cf317f10bb0b6" exitCode=0 Jul 28 13:26:34.064411 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:34.064257 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" event={"ID":"9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab","Type":"ContainerDied","Data":"f820719f646fe39259122c77de1b4085207d1630df8d334bba6cf317f10bb0b6"} Jul 28 13:26:34.064411 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:34.064275 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" Jul 28 13:26:34.064411 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:34.064301 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vdv2d" event={"ID":"9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab","Type":"ContainerDied","Data":"415a1ea308bbe8df8022e3a34674e3f3188835208b3ca57028d408fa96fb6b31"} Jul 28 13:26:34.064411 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:34.064323 2573 scope.go:117] "RemoveContainer" containerID="f820719f646fe39259122c77de1b4085207d1630df8d334bba6cf317f10bb0b6" Jul 28 13:26:34.073987 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:34.073971 2573 scope.go:117] "RemoveContainer" containerID="f820719f646fe39259122c77de1b4085207d1630df8d334bba6cf317f10bb0b6" Jul 28 13:26:34.074269 ip-10-0-136-145 kubenswrapper[2573]: E0728 13:26:34.074251 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f820719f646fe39259122c77de1b4085207d1630df8d334bba6cf317f10bb0b6\": container with ID starting with f820719f646fe39259122c77de1b4085207d1630df8d334bba6cf317f10bb0b6 not found: ID does not exist" containerID="f820719f646fe39259122c77de1b4085207d1630df8d334bba6cf317f10bb0b6" Jul 28 13:26:34.074327 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:34.074279 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f820719f646fe39259122c77de1b4085207d1630df8d334bba6cf317f10bb0b6"} err="failed to get container status \"f820719f646fe39259122c77de1b4085207d1630df8d334bba6cf317f10bb0b6\": rpc error: code = NotFound desc = could not find container \"f820719f646fe39259122c77de1b4085207d1630df8d334bba6cf317f10bb0b6\": container with ID starting with f820719f646fe39259122c77de1b4085207d1630df8d334bba6cf317f10bb0b6 not found: ID does not exist" Jul 28 13:26:34.085401 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:34.085373 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vdv2d"] Jul 28 13:26:34.088472 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:34.088433 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vdv2d"] Jul 28 13:26:35.268035 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:35.268004 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab" path="/var/lib/kubelet/pods/9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab/volumes" Jul 28 13:26:47.625197 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.625133 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" podUID="07e2500f-d2d1-4234-ab6e-1c5b726d2ce5" containerName="main" containerID="cri-o://99a240523893cf9e9547a5aecacd0410d4b3015425d6be65912351426120170c" gracePeriod=45 Jul 28 13:26:47.866127 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.866103 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:26:47.983968 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.983931 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-dshm\") pod \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " Jul 28 13:26:47.984166 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.983977 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-model-cache\") pod \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " Jul 28 13:26:47.984166 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.984002 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-tmp-dir\") pod \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " Jul 28 13:26:47.984166 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.984045 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-tls-certs\") pod \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " Jul 28 13:26:47.984166 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.984102 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-home\") pod \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " Jul 28 13:26:47.984166 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.984126 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm44w\" (UniqueName: \"kubernetes.io/projected/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-kube-api-access-mm44w\") pod \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\" (UID: \"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5\") " Jul 28 13:26:47.984402 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.984256 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-model-cache" (OuterVolumeSpecName: "model-cache") pod "07e2500f-d2d1-4234-ab6e-1c5b726d2ce5" (UID: "07e2500f-d2d1-4234-ab6e-1c5b726d2ce5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:26:47.984402 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.984353 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-home" (OuterVolumeSpecName: "home") pod "07e2500f-d2d1-4234-ab6e-1c5b726d2ce5" (UID: "07e2500f-d2d1-4234-ab6e-1c5b726d2ce5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:26:47.984402 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.984383 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-model-cache\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:26:47.984566 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.984470 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07e2500f-d2d1-4234-ab6e-1c5b726d2ce5" (UID: "07e2500f-d2d1-4234-ab6e-1c5b726d2ce5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:26:47.986471 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.986449 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-dshm" (OuterVolumeSpecName: "dshm") pod "07e2500f-d2d1-4234-ab6e-1c5b726d2ce5" (UID: "07e2500f-d2d1-4234-ab6e-1c5b726d2ce5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:26:47.986561 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.986503 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07e2500f-d2d1-4234-ab6e-1c5b726d2ce5" (UID: "07e2500f-d2d1-4234-ab6e-1c5b726d2ce5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:26:47.986660 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:47.986640 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-kube-api-access-mm44w" (OuterVolumeSpecName: "kube-api-access-mm44w") pod "07e2500f-d2d1-4234-ab6e-1c5b726d2ce5" (UID: "07e2500f-d2d1-4234-ab6e-1c5b726d2ce5"). InnerVolumeSpecName "kube-api-access-mm44w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:26:48.085496 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.085414 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-tmp-dir\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:26:48.085496 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.085490 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-tls-certs\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:26:48.085496 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.085503 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-home\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:26:48.085728 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.085515 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm44w\" (UniqueName: \"kubernetes.io/projected/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-kube-api-access-mm44w\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:26:48.085728 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.085528 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5-dshm\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:26:48.145940 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.145906 2573 generic.go:356] "Generic (PLEG): container finished" podID="07e2500f-d2d1-4234-ab6e-1c5b726d2ce5" containerID="99a240523893cf9e9547a5aecacd0410d4b3015425d6be65912351426120170c" exitCode=0 Jul 28 13:26:48.146116 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.145947 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" event={"ID":"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5","Type":"ContainerDied","Data":"99a240523893cf9e9547a5aecacd0410d4b3015425d6be65912351426120170c"} Jul 28 13:26:48.146116 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.145954 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" Jul 28 13:26:48.146116 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.145973 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x" event={"ID":"07e2500f-d2d1-4234-ab6e-1c5b726d2ce5","Type":"ContainerDied","Data":"6c46da304ef810366955dc3e1641d7908548b085f1ba48e2be7f0582d7c753ad"} Jul 28 13:26:48.146116 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.145989 2573 scope.go:117] "RemoveContainer" containerID="99a240523893cf9e9547a5aecacd0410d4b3015425d6be65912351426120170c" Jul 28 13:26:48.155920 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.155900 2573 scope.go:117] "RemoveContainer" containerID="99a240523893cf9e9547a5aecacd0410d4b3015425d6be65912351426120170c" Jul 28 13:26:48.156169 ip-10-0-136-145 kubenswrapper[2573]: E0728 13:26:48.156151 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a240523893cf9e9547a5aecacd0410d4b3015425d6be65912351426120170c\": container with ID starting with 99a240523893cf9e9547a5aecacd0410d4b3015425d6be65912351426120170c not found: ID does not exist" containerID="99a240523893cf9e9547a5aecacd0410d4b3015425d6be65912351426120170c" Jul 28 13:26:48.156208 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.156180 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a240523893cf9e9547a5aecacd0410d4b3015425d6be65912351426120170c"} err="failed to get container status \"99a240523893cf9e9547a5aecacd0410d4b3015425d6be65912351426120170c\": rpc error: code = NotFound desc = could not find container \"99a240523893cf9e9547a5aecacd0410d4b3015425d6be65912351426120170c\": container with ID starting with 99a240523893cf9e9547a5aecacd0410d4b3015425d6be65912351426120170c not found: ID does not exist" Jul 28 13:26:48.167811 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.167785 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x"] Jul 28 13:26:48.171217 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:48.171195 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-8fb56b/ipp-route-8fb56b-kserve-79c9dcb747-wgr8x"] Jul 28 13:26:49.268260 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:26:49.268226 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e2500f-d2d1-4234-ab6e-1c5b726d2ce5" path="/var/lib/kubelet/pods/07e2500f-d2d1-4234-ab6e-1c5b726d2ce5/volumes" Jul 28 13:27:22.996287 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:22.996253 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns"] Jul 28 13:27:22.996769 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:22.996588 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab" containerName="registry-server" Jul 28 13:27:22.996769 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:22.996600 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab" containerName="registry-server" Jul 28 13:27:22.996769 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:22.996618 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e2500f-d2d1-4234-ab6e-1c5b726d2ce5" containerName="main" Jul 28 13:27:22.996769 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:22.996624 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e2500f-d2d1-4234-ab6e-1c5b726d2ce5" containerName="main" Jul 28 13:27:22.996769 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:22.996689 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d5c8c83-7ccd-4d2b-b48c-8ff6ad3540ab" containerName="registry-server" Jul 28 13:27:22.996769 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:22.996699 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e2500f-d2d1-4234-ab6e-1c5b726d2ce5" containerName="main" Jul 28 13:27:23.001180 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:23.001159 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns" Jul 28 13:27:23.004410 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:23.004392 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:27:23.009560 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:23.009535 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns"] Jul 28 13:27:23.079255 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:23.079221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc9aa15-e0ba-4408-8d1f-387cb49b5593-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns\" (UID: \"1cc9aa15-e0ba-4408-8d1f-387cb49b5593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns" Jul 28 13:27:23.180198 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:23.180166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc9aa15-e0ba-4408-8d1f-387cb49b5593-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns\" (UID: \"1cc9aa15-e0ba-4408-8d1f-387cb49b5593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns" Jul 28 13:27:23.180874 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:23.180854 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc9aa15-e0ba-4408-8d1f-387cb49b5593-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns\" (UID: \"1cc9aa15-e0ba-4408-8d1f-387cb49b5593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns" Jul 28 13:27:23.313190 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:23.313109 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns" Jul 28 13:27:23.463954 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:23.463926 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns"] Jul 28 13:27:23.465456 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:27:23.465401 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc9aa15_e0ba_4408_8d1f_387cb49b5593.slice/crio-30a9f6ae6357c15000f9eb10292d87ab7632f3c050191688496ed871cd38fa71 WatchSource:0}: Error finding container 30a9f6ae6357c15000f9eb10292d87ab7632f3c050191688496ed871cd38fa71: Status 404 returned error can't find the container with id 30a9f6ae6357c15000f9eb10292d87ab7632f3c050191688496ed871cd38fa71 Jul 28 13:27:24.342247 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:24.342208 2573 generic.go:356] "Generic (PLEG): container finished" podID="1cc9aa15-e0ba-4408-8d1f-387cb49b5593" containerID="e65c0572d9ed65269751591dd071c48510dd9e3ca0fa5d50bbea4f4f8725941c" exitCode=0 Jul 28 13:27:24.342669 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:24.342261 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns" event={"ID":"1cc9aa15-e0ba-4408-8d1f-387cb49b5593","Type":"ContainerDied","Data":"e65c0572d9ed65269751591dd071c48510dd9e3ca0fa5d50bbea4f4f8725941c"} Jul 28 13:27:24.342669 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:24.342315 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns" event={"ID":"1cc9aa15-e0ba-4408-8d1f-387cb49b5593","Type":"ContainerStarted","Data":"30a9f6ae6357c15000f9eb10292d87ab7632f3c050191688496ed871cd38fa71"} Jul 28 13:27:25.488429 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:25.488399 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns" Jul 28 13:27:25.604264 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:25.604177 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc9aa15-e0ba-4408-8d1f-387cb49b5593-maas-api-service-ca\") pod \"1cc9aa15-e0ba-4408-8d1f-387cb49b5593\" (UID: \"1cc9aa15-e0ba-4408-8d1f-387cb49b5593\") " Jul 28 13:27:25.604617 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:25.604595 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc9aa15-e0ba-4408-8d1f-387cb49b5593-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1cc9aa15-e0ba-4408-8d1f-387cb49b5593" (UID: "1cc9aa15-e0ba-4408-8d1f-387cb49b5593"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:27:25.705693 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:25.705660 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc9aa15-e0ba-4408-8d1f-387cb49b5593-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:27:26.355036 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:26.355005 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns" Jul 28 13:27:26.355239 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:26.355042 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns" event={"ID":"1cc9aa15-e0ba-4408-8d1f-387cb49b5593","Type":"ContainerDied","Data":"30a9f6ae6357c15000f9eb10292d87ab7632f3c050191688496ed871cd38fa71"} Jul 28 13:27:26.355239 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:26.355101 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30a9f6ae6357c15000f9eb10292d87ab7632f3c050191688496ed871cd38fa71" Jul 28 13:27:33.449420 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:33.449364 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz"] Jul 28 13:27:33.449902 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:33.449781 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" podUID="5fd84bd6-b977-427e-9c1e-acf2da21a36d" containerName="maas-api" containerID="cri-o://7d295cca4f051acac0238a01a261e77b0dfe0f9e73970530739de542304c0fe1" gracePeriod=30 Jul 28 13:27:33.709815 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:33.709740 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:27:33.778666 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:33.778630 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd84bd6-b977-427e-9c1e-acf2da21a36d-maas-api-tls\") pod \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\" (UID: \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\") " Jul 28 13:27:33.778863 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:33.778679 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd84bd6-b977-427e-9c1e-acf2da21a36d-openshift-service-ca\") pod \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\" (UID: \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\") " Jul 28 13:27:33.778863 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:33.778708 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzcdj\" (UniqueName: \"kubernetes.io/projected/5fd84bd6-b977-427e-9c1e-acf2da21a36d-kube-api-access-kzcdj\") pod \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\" (UID: \"5fd84bd6-b977-427e-9c1e-acf2da21a36d\") " Jul 28 13:27:33.779018 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:33.778995 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd84bd6-b977-427e-9c1e-acf2da21a36d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5fd84bd6-b977-427e-9c1e-acf2da21a36d" (UID: "5fd84bd6-b977-427e-9c1e-acf2da21a36d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:27:33.780941 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:33.780910 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd84bd6-b977-427e-9c1e-acf2da21a36d-kube-api-access-kzcdj" (OuterVolumeSpecName: "kube-api-access-kzcdj") pod "5fd84bd6-b977-427e-9c1e-acf2da21a36d" (UID: "5fd84bd6-b977-427e-9c1e-acf2da21a36d"). InnerVolumeSpecName "kube-api-access-kzcdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:27:33.780941 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:33.780935 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd84bd6-b977-427e-9c1e-acf2da21a36d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5fd84bd6-b977-427e-9c1e-acf2da21a36d" (UID: "5fd84bd6-b977-427e-9c1e-acf2da21a36d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:27:33.879724 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:33.879690 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd84bd6-b977-427e-9c1e-acf2da21a36d-maas-api-tls\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:27:33.879724 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:33.879718 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd84bd6-b977-427e-9c1e-acf2da21a36d-openshift-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:27:33.879724 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:33.879728 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzcdj\" (UniqueName: \"kubernetes.io/projected/5fd84bd6-b977-427e-9c1e-acf2da21a36d-kube-api-access-kzcdj\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:27:34.402199 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:34.402161 2573 generic.go:356] "Generic (PLEG): container finished" podID="5fd84bd6-b977-427e-9c1e-acf2da21a36d" containerID="7d295cca4f051acac0238a01a261e77b0dfe0f9e73970530739de542304c0fe1" exitCode=0 Jul 28 13:27:34.402390 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:34.402199 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" event={"ID":"5fd84bd6-b977-427e-9c1e-acf2da21a36d","Type":"ContainerDied","Data":"7d295cca4f051acac0238a01a261e77b0dfe0f9e73970530739de542304c0fe1"} Jul 28 13:27:34.402390 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:34.402252 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" event={"ID":"5fd84bd6-b977-427e-9c1e-acf2da21a36d","Type":"ContainerDied","Data":"a896e3b0d3aba651d305219dc9f367a5f70c904a51f6e9ff5748a6c630bfe931"} Jul 28 13:27:34.402390 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:34.402274 2573 scope.go:117] "RemoveContainer" containerID="7d295cca4f051acac0238a01a261e77b0dfe0f9e73970530739de542304c0fe1" Jul 28 13:27:34.402390 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:34.402210 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz" Jul 28 13:27:34.412143 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:34.412123 2573 scope.go:117] "RemoveContainer" containerID="7d295cca4f051acac0238a01a261e77b0dfe0f9e73970530739de542304c0fe1" Jul 28 13:27:34.412413 ip-10-0-136-145 kubenswrapper[2573]: E0728 13:27:34.412395 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d295cca4f051acac0238a01a261e77b0dfe0f9e73970530739de542304c0fe1\": container with ID starting with 7d295cca4f051acac0238a01a261e77b0dfe0f9e73970530739de542304c0fe1 not found: ID does not exist" containerID="7d295cca4f051acac0238a01a261e77b0dfe0f9e73970530739de542304c0fe1" Jul 28 13:27:34.412483 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:34.412423 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d295cca4f051acac0238a01a261e77b0dfe0f9e73970530739de542304c0fe1"} err="failed to get container status \"7d295cca4f051acac0238a01a261e77b0dfe0f9e73970530739de542304c0fe1\": rpc error: code = NotFound desc = could not find container \"7d295cca4f051acac0238a01a261e77b0dfe0f9e73970530739de542304c0fe1\": container with ID starting with 7d295cca4f051acac0238a01a261e77b0dfe0f9e73970530739de542304c0fe1 not found: ID does not exist" Jul 28 13:27:34.424950 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:34.424922 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz"] Jul 28 13:27:34.429063 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:34.429036 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-06a4d2-74dc8b84ff-9xqbz"] Jul 28 13:27:35.267599 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:35.267565 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fd84bd6-b977-427e-9c1e-acf2da21a36d" path="/var/lib/kubelet/pods/5fd84bd6-b977-427e-9c1e-acf2da21a36d/volumes" Jul 28 13:27:36.832987 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:36.832951 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:27:38.071203 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:38.071163 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns"] Jul 28 13:27:38.074894 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:38.074855 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06a4d2-22nns"] Jul 28 13:27:39.266925 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:39.266890 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc9aa15-e0ba-4408-8d1f-387cb49b5593" path="/var/lib/kubelet/pods/1cc9aa15-e0ba-4408-8d1f-387cb49b5593/volumes" Jul 28 13:27:46.098552 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.098514 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx"] Jul 28 13:27:46.098932 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.098860 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc9aa15-e0ba-4408-8d1f-387cb49b5593" containerName="revoke-keys" Jul 28 13:27:46.098932 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.098875 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc9aa15-e0ba-4408-8d1f-387cb49b5593" containerName="revoke-keys" Jul 28 13:27:46.098932 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.098904 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd84bd6-b977-427e-9c1e-acf2da21a36d" containerName="maas-api" Jul 28 13:27:46.098932 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.098909 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd84bd6-b977-427e-9c1e-acf2da21a36d" containerName="maas-api" Jul 28 13:27:46.099056 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.098972 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc9aa15-e0ba-4408-8d1f-387cb49b5593" containerName="revoke-keys" Jul 28 13:27:46.099056 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.098984 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd84bd6-b977-427e-9c1e-acf2da21a36d" containerName="maas-api" Jul 28 13:27:46.103370 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.103351 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx" Jul 28 13:27:46.118130 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.118105 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:27:46.120591 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.120573 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 13:27:46.130496 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.130466 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx"] Jul 28 13:27:46.289372 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.289327 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d16c51-c017-44c2-9657-c2f29da30457-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx\" (UID: \"01d16c51-c017-44c2-9657-c2f29da30457\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx" Jul 28 13:27:46.390422 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.390390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d16c51-c017-44c2-9657-c2f29da30457-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx\" (UID: \"01d16c51-c017-44c2-9657-c2f29da30457\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx" Jul 28 13:27:46.391051 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.391024 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d16c51-c017-44c2-9657-c2f29da30457-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx\" (UID: \"01d16c51-c017-44c2-9657-c2f29da30457\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx" Jul 28 13:27:46.413044 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.412990 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx" Jul 28 13:27:46.568792 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:46.568769 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx"] Jul 28 13:27:46.570128 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:27:46.570102 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d16c51_c017_44c2_9657_c2f29da30457.slice/crio-b97e6b91483b758be1c92803c3ac89ffa5e75749941ebefa0fbde1528b8515a5 WatchSource:0}: Error finding container b97e6b91483b758be1c92803c3ac89ffa5e75749941ebefa0fbde1528b8515a5: Status 404 returned error can't find the container with id b97e6b91483b758be1c92803c3ac89ffa5e75749941ebefa0fbde1528b8515a5 Jul 28 13:27:47.476479 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:47.476420 2573 generic.go:356] "Generic (PLEG): container finished" podID="01d16c51-c017-44c2-9657-c2f29da30457" containerID="526614797ae65c590548b5cfa9d36c43ee8175ebbc7c955487817b2018f1619d" exitCode=0 Jul 28 13:27:47.476479 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:47.476471 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx" event={"ID":"01d16c51-c017-44c2-9657-c2f29da30457","Type":"ContainerDied","Data":"526614797ae65c590548b5cfa9d36c43ee8175ebbc7c955487817b2018f1619d"} Jul 28 13:27:47.476906 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:47.476531 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx" event={"ID":"01d16c51-c017-44c2-9657-c2f29da30457","Type":"ContainerStarted","Data":"b97e6b91483b758be1c92803c3ac89ffa5e75749941ebefa0fbde1528b8515a5"} Jul 28 13:27:48.611902 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:48.611871 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx" Jul 28 13:27:48.714580 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:48.714545 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d16c51-c017-44c2-9657-c2f29da30457-maas-api-service-ca\") pod \"01d16c51-c017-44c2-9657-c2f29da30457\" (UID: \"01d16c51-c017-44c2-9657-c2f29da30457\") " Jul 28 13:27:48.714911 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:48.714887 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d16c51-c017-44c2-9657-c2f29da30457-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "01d16c51-c017-44c2-9657-c2f29da30457" (UID: "01d16c51-c017-44c2-9657-c2f29da30457"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:27:48.816231 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:48.816144 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d16c51-c017-44c2-9657-c2f29da30457-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:27:49.489146 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:49.489109 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx" event={"ID":"01d16c51-c017-44c2-9657-c2f29da30457","Type":"ContainerDied","Data":"b97e6b91483b758be1c92803c3ac89ffa5e75749941ebefa0fbde1528b8515a5"} Jul 28 13:27:49.489323 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:49.489161 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx" Jul 28 13:27:49.489323 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:49.489170 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b97e6b91483b758be1c92803c3ac89ffa5e75749941ebefa0fbde1528b8515a5" Jul 28 13:27:56.668150 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:56.668114 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-67d68f-29754075-8q99f"] Jul 28 13:27:56.670285 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:56.670262 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-67d68f-29754075-8q99f"] Jul 28 13:27:57.268589 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:57.268558 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad589737-8064-4393-92c7-89de51fc8703" path="/var/lib/kubelet/pods/ad589737-8064-4393-92c7-89de51fc8703/volumes" Jul 28 13:27:59.630020 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:27:59.629981 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:28:01.132594 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:01.132539 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx"] Jul 28 13:28:01.135267 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:01.135232 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-67d68f-ldtqx"] Jul 28 13:28:01.267879 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:01.267847 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d16c51-c017-44c2-9657-c2f29da30457" path="/var/lib/kubelet/pods/01d16c51-c017-44c2-9657-c2f29da30457/volumes" Jul 28 13:28:08.215898 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:08.215861 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn"] Jul 28 13:28:08.216272 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:08.216193 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d16c51-c017-44c2-9657-c2f29da30457" containerName="revoke-keys" Jul 28 13:28:08.216272 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:08.216204 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d16c51-c017-44c2-9657-c2f29da30457" containerName="revoke-keys" Jul 28 13:28:08.216272 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:08.216271 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d16c51-c017-44c2-9657-c2f29da30457" containerName="revoke-keys" Jul 28 13:28:08.219316 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:08.219299 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn" Jul 28 13:28:08.222181 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:08.222165 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 13:28:08.222358 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:08.222342 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5hqn\"" Jul 28 13:28:08.226884 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:08.226854 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn"] Jul 28 13:28:08.278755 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:08.278719 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab35659-787a-4a2c-b12d-099904710877-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-827464-lmnkn\" (UID: \"dab35659-787a-4a2c-b12d-099904710877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn" Jul 28 13:28:08.380335 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:08.379872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab35659-787a-4a2c-b12d-099904710877-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-827464-lmnkn\" (UID: \"dab35659-787a-4a2c-b12d-099904710877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn" Jul 28 13:28:08.384935 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:08.380956 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab35659-787a-4a2c-b12d-099904710877-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-827464-lmnkn\" (UID: \"dab35659-787a-4a2c-b12d-099904710877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn" Jul 28 13:28:08.531549 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:08.531491 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn" Jul 28 13:28:08.664880 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:08.664856 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn"] Jul 28 13:28:08.667365 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:28:08.667339 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab35659_787a_4a2c_b12d_099904710877.slice/crio-605208a443d2e3ee50fc6114a85ceed6546d0f1cec3f764e01888a65a161539e WatchSource:0}: Error finding container 605208a443d2e3ee50fc6114a85ceed6546d0f1cec3f764e01888a65a161539e: Status 404 returned error can't find the container with id 605208a443d2e3ee50fc6114a85ceed6546d0f1cec3f764e01888a65a161539e Jul 28 13:28:09.606472 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:09.606380 2573 generic.go:356] "Generic (PLEG): container finished" podID="dab35659-787a-4a2c-b12d-099904710877" containerID="465a5a6606b4e580949e228f06fd7ae036038441d708506cb6031b580db13ba5" exitCode=0 Jul 28 13:28:09.606472 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:09.606420 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn" event={"ID":"dab35659-787a-4a2c-b12d-099904710877","Type":"ContainerDied","Data":"465a5a6606b4e580949e228f06fd7ae036038441d708506cb6031b580db13ba5"} Jul 28 13:28:09.606854 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:09.606474 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn" event={"ID":"dab35659-787a-4a2c-b12d-099904710877","Type":"ContainerStarted","Data":"605208a443d2e3ee50fc6114a85ceed6546d0f1cec3f764e01888a65a161539e"} Jul 28 13:28:10.744613 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:10.744590 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn" Jul 28 13:28:10.799057 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:10.799028 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab35659-787a-4a2c-b12d-099904710877-maas-api-service-ca\") pod \"dab35659-787a-4a2c-b12d-099904710877\" (UID: \"dab35659-787a-4a2c-b12d-099904710877\") " Jul 28 13:28:10.799347 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:10.799324 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab35659-787a-4a2c-b12d-099904710877-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dab35659-787a-4a2c-b12d-099904710877" (UID: "dab35659-787a-4a2c-b12d-099904710877"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:28:10.900608 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:10.900586 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab35659-787a-4a2c-b12d-099904710877-maas-api-service-ca\") on node \"ip-10-0-136-145.ec2.internal\" DevicePath \"\"" Jul 28 13:28:11.618493 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:11.618409 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn" Jul 28 13:28:11.618493 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:11.618420 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn" event={"ID":"dab35659-787a-4a2c-b12d-099904710877","Type":"ContainerDied","Data":"605208a443d2e3ee50fc6114a85ceed6546d0f1cec3f764e01888a65a161539e"} Jul 28 13:28:11.618493 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:11.618486 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="605208a443d2e3ee50fc6114a85ceed6546d0f1cec3f764e01888a65a161539e" Jul 28 13:28:15.689228 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:15.689199 2573 scope.go:117] "RemoveContainer" containerID="57de88a49d3885f562319bcc52615a42187d9c04a5a43f2489616537947b25b4" Jul 28 13:28:18.597428 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:18.597392 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-827464-29754075-vztfq"] Jul 28 13:28:18.601750 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:18.601723 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-827464-29754075-vztfq"] Jul 28 13:28:19.267558 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:19.267528 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e856e63d-1826-42d1-baf4-68d5723d93de" path="/var/lib/kubelet/pods/e856e63d-1826-42d1-baf4-68d5723d93de/volumes" Jul 28 13:28:20.733218 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:20.733175 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5b8ps"] Jul 28 13:28:23.296956 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:23.296922 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn"] Jul 28 13:28:23.300754 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:23.300728 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-827464-lmnkn"] Jul 28 13:28:25.268275 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:25.268243 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab35659-787a-4a2c-b12d-099904710877" path="/var/lib/kubelet/pods/dab35659-787a-4a2c-b12d-099904710877/volumes" Jul 28 13:28:37.646119 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:37.646039 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7974886bdb-72sjs_0dd05d93-2a5f-41c7-8996-e9560b576309/copy-manifests/0.log" Jul 28 13:28:37.665335 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:37.665301 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7974886bdb-72sjs_0dd05d93-2a5f-41c7-8996-e9560b576309/manager/0.log" Jul 28 13:28:38.603694 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:38.603665 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754075-25t7m_a27a355a-5dfe-4674-9e38-864f49835ef6/cleanup/0.log" Jul 28 13:28:39.612302 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:39.612271 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj_dce6f400-0ed3-4fdc-8916-a970b4aeca78/extract/0.log" Jul 28 13:28:39.618682 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:39.618656 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj_dce6f400-0ed3-4fdc-8916-a970b4aeca78/util/0.log" Jul 28 13:28:39.625564 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:39.625542 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj_dce6f400-0ed3-4fdc-8916-a970b4aeca78/pull/0.log" Jul 28 13:28:39.748393 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:39.748371 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf_1a9b484d-2f1a-4429-97b9-a13df4c3d9d3/util/0.log" Jul 28 13:28:39.754820 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:39.754802 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf_1a9b484d-2f1a-4429-97b9-a13df4c3d9d3/pull/0.log" Jul 28 13:28:39.761817 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:39.761800 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf_1a9b484d-2f1a-4429-97b9-a13df4c3d9d3/extract/0.log" Jul 28 13:28:39.896756 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:39.896730 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn_e0ca90fd-3305-453e-a8da-c322bf163a6f/extract/0.log" Jul 28 13:28:39.902900 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:39.902873 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn_e0ca90fd-3305-453e-a8da-c322bf163a6f/util/0.log" Jul 28 13:28:39.909111 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:39.909087 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn_e0ca90fd-3305-453e-a8da-c322bf163a6f/pull/0.log" Jul 28 13:28:40.039768 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:40.039633 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs_7e4b1835-079b-4e48-88df-794410cb2261/util/0.log" Jul 28 13:28:40.046088 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:40.046057 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs_7e4b1835-079b-4e48-88df-794410cb2261/pull/0.log" Jul 28 13:28:40.052109 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:40.052080 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs_7e4b1835-079b-4e48-88df-794410cb2261/extract/0.log" Jul 28 13:28:41.051759 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:41.051733 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-5b8ps_e9e35209-fe83-4acb-9140-2e70b8f129b3/limitador/0.log" Jul 28 13:28:41.757117 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:41.757085 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jqjlx_b20ed721-ef7e-47eb-ad07-1f0a28163f89/discovery/0.log" Jul 28 13:28:43.026720 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:43.026680 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-x5fk7_f90f3bdd-ec45-4ab6-bc41-363729f69bb1/main/0.log" Jul 28 13:28:43.563689 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:43.563664 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-sswpn_4ab58fc0-b4b2-497f-bb79-07972d507846/main/0.log" Jul 28 13:28:53.325496 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:53.325460 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cg9km_bbf8d227-a317-4952-a0b2-7ca6d602d804/global-pull-secret-syncer/0.log" Jul 28 13:28:53.512573 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:53.512545 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n88bh_c7d55ba4-f5a8-48fd-8c9f-932ceaedc498/konnectivity-agent/0.log" Jul 28 13:28:53.631230 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:53.631161 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-145.ec2.internal_4541c39d7375e0d14063052c99c9a71f/haproxy/0.log" Jul 28 13:28:58.759204 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:58.759157 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj_dce6f400-0ed3-4fdc-8916-a970b4aeca78/extract/0.log" Jul 28 13:28:58.784426 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:58.784393 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj_dce6f400-0ed3-4fdc-8916-a970b4aeca78/util/0.log" Jul 28 13:28:58.812557 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:58.812525 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759flxvj_dce6f400-0ed3-4fdc-8916-a970b4aeca78/pull/0.log" Jul 28 13:28:58.859414 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:58.859389 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf_1a9b484d-2f1a-4429-97b9-a13df4c3d9d3/extract/0.log" Jul 28 13:28:58.887456 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:58.887416 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf_1a9b484d-2f1a-4429-97b9-a13df4c3d9d3/util/0.log" Jul 28 13:28:58.924018 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:58.923985 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08gzsf_1a9b484d-2f1a-4429-97b9-a13df4c3d9d3/pull/0.log" Jul 28 13:28:58.956531 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:58.956507 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn_e0ca90fd-3305-453e-a8da-c322bf163a6f/extract/0.log" Jul 28 13:28:58.979216 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:58.979196 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn_e0ca90fd-3305-453e-a8da-c322bf163a6f/util/0.log" Jul 28 13:28:59.000333 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:59.000314 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hxvkn_e0ca90fd-3305-453e-a8da-c322bf163a6f/pull/0.log" Jul 28 13:28:59.030487 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:59.030414 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs_7e4b1835-079b-4e48-88df-794410cb2261/extract/0.log" Jul 28 13:28:59.055158 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:59.055130 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs_7e4b1835-079b-4e48-88df-794410cb2261/util/0.log" Jul 28 13:28:59.077940 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:28:59.077921 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17lnfs_7e4b1835-079b-4e48-88df-794410cb2261/pull/0.log" Jul 28 13:29:00.168108 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:00.168076 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-5b8ps_e9e35209-fe83-4acb-9140-2e70b8f129b3/limitador/0.log" Jul 28 13:29:02.199249 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:02.199219 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-ldvw4_9556f668-494f-4d33-8bc4-a3e615e50f52/cluster-monitoring-operator/0.log" Jul 28 13:29:02.365500 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:02.365472 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8rssd_5302424c-8d67-4228-a781-d3b6aeab2288/node-exporter/0.log" Jul 28 13:29:02.387075 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:02.387049 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8rssd_5302424c-8d67-4228-a781-d3b6aeab2288/kube-rbac-proxy/0.log" Jul 28 13:29:02.415152 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:02.415128 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8rssd_5302424c-8d67-4228-a781-d3b6aeab2288/init-textfile/0.log" Jul 28 13:29:02.976639 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:02.976614 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-8njsx_993c23fc-72c5-4a38-a973-41c653c2e958/prometheus-operator-admission-webhook/0.log" Jul 28 13:29:03.006988 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.006957 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dbcbd6cf6-djpjq_cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb/telemeter-client/0.log" Jul 28 13:29:03.029591 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.029570 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dbcbd6cf6-djpjq_cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb/reload/0.log" Jul 28 13:29:03.050614 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.050591 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dbcbd6cf6-djpjq_cc1093eb-6e7c-4fae-9b9c-e0ed034c7acb/kube-rbac-proxy/0.log" Jul 28 13:29:03.078518 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.078494 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ffb65cc44-wt7jb_50ceed3d-85bc-4237-a07e-6141140d1f72/thanos-query/0.log" Jul 28 13:29:03.101264 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.101219 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ffb65cc44-wt7jb_50ceed3d-85bc-4237-a07e-6141140d1f72/kube-rbac-proxy-web/0.log" Jul 28 13:29:03.122808 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.122791 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ffb65cc44-wt7jb_50ceed3d-85bc-4237-a07e-6141140d1f72/kube-rbac-proxy/0.log" Jul 28 13:29:03.146847 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.146827 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ffb65cc44-wt7jb_50ceed3d-85bc-4237-a07e-6141140d1f72/prom-label-proxy/0.log" Jul 28 13:29:03.171877 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.171858 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ffb65cc44-wt7jb_50ceed3d-85bc-4237-a07e-6141140d1f72/kube-rbac-proxy-rules/0.log" Jul 28 13:29:03.200817 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.200792 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ffb65cc44-wt7jb_50ceed3d-85bc-4237-a07e-6141140d1f72/kube-rbac-proxy-metrics/0.log" Jul 28 13:29:03.943123 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.943088 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724"] Jul 28 13:29:03.943461 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.943425 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab35659-787a-4a2c-b12d-099904710877" containerName="revoke-keys" Jul 28 13:29:03.943461 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.943453 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab35659-787a-4a2c-b12d-099904710877" containerName="revoke-keys" Jul 28 13:29:03.943558 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.943515 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab35659-787a-4a2c-b12d-099904710877" containerName="revoke-keys" Jul 28 13:29:03.946316 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.946301 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:03.948998 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.948968 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5hhgv\"/\"openshift-service-ca.crt\"" Jul 28 13:29:03.949103 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.948971 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5hhgv\"/\"kube-root-ca.crt\"" Jul 28 13:29:03.950664 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.950644 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5hhgv\"/\"default-dockercfg-tmkdf\"" Jul 28 13:29:03.952983 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:03.952959 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724"] Jul 28 13:29:04.025101 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.025071 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae248978-aa1c-4c4a-ba4c-6155f0205934-lib-modules\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.025283 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.025110 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae248978-aa1c-4c4a-ba4c-6155f0205934-podres\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.025283 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.025134 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae248978-aa1c-4c4a-ba4c-6155f0205934-sys\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.025283 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.025219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae248978-aa1c-4c4a-ba4c-6155f0205934-proc\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.025283 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.025258 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cc2l\" (UniqueName: \"kubernetes.io/projected/ae248978-aa1c-4c4a-ba4c-6155f0205934-kube-api-access-4cc2l\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.125953 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.125921 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cc2l\" (UniqueName: \"kubernetes.io/projected/ae248978-aa1c-4c4a-ba4c-6155f0205934-kube-api-access-4cc2l\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.126115 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.125968 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae248978-aa1c-4c4a-ba4c-6155f0205934-lib-modules\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.126115 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.126074 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae248978-aa1c-4c4a-ba4c-6155f0205934-lib-modules\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.126115 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.126077 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae248978-aa1c-4c4a-ba4c-6155f0205934-podres\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.126235 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.126115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae248978-aa1c-4c4a-ba4c-6155f0205934-sys\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.126235 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.126161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae248978-aa1c-4c4a-ba4c-6155f0205934-proc\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.126235 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.126193 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae248978-aa1c-4c4a-ba4c-6155f0205934-sys\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.126235 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.126220 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae248978-aa1c-4c4a-ba4c-6155f0205934-podres\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.126373 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.126235 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae248978-aa1c-4c4a-ba4c-6155f0205934-proc\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.133730 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.133702 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cc2l\" (UniqueName: \"kubernetes.io/projected/ae248978-aa1c-4c4a-ba4c-6155f0205934-kube-api-access-4cc2l\") pod \"perf-node-gather-daemonset-bs724\" (UID: \"ae248978-aa1c-4c4a-ba4c-6155f0205934\") " pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.257976 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.257896 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.393498 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.393472 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724"] Jul 28 13:29:04.395026 ip-10-0-136-145 kubenswrapper[2573]: W0728 13:29:04.395000 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podae248978_aa1c_4c4a_ba4c_6155f0205934.slice/crio-c21757e0b937d02313891576e843fdaefb5cdae6baf66fbfc7aa603abb3d421f WatchSource:0}: Error finding container c21757e0b937d02313891576e843fdaefb5cdae6baf66fbfc7aa603abb3d421f: Status 404 returned error can't find the container with id c21757e0b937d02313891576e843fdaefb5cdae6baf66fbfc7aa603abb3d421f Jul 28 13:29:04.921202 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.921168 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" event={"ID":"ae248978-aa1c-4c4a-ba4c-6155f0205934","Type":"ContainerStarted","Data":"495763b58394e1d4b7d9d4f263e094f2aa81261d3da267fe43cb14e5eec9f1a1"} Jul 28 13:29:04.921388 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.921217 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" event={"ID":"ae248978-aa1c-4c4a-ba4c-6155f0205934","Type":"ContainerStarted","Data":"c21757e0b937d02313891576e843fdaefb5cdae6baf66fbfc7aa603abb3d421f"} Jul 28 13:29:04.921388 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.921252 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:04.938823 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:04.938783 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" podStartSLOduration=1.938769061 podStartE2EDuration="1.938769061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:29:04.936254355 +0000 UTC m=+3530.195545095" watchObservedRunningTime="2026-07-28 13:29:04.938769061 +0000 UTC m=+3530.198059801" Jul 28 13:29:05.436786 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:05.436749 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-57886bb876-gq5dz_52c70995-54d6-41b2-8e0c-2b08959bfe15/console/0.log" Jul 28 13:29:07.171551 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:07.171517 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mfjqx_bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4/dns/0.log" Jul 28 13:29:07.194894 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:07.194868 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mfjqx_bf4637d3-b8e1-4a3c-9f5b-8445e881c4c4/kube-rbac-proxy/0.log" Jul 28 13:29:07.351463 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:07.351416 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wd7wp_8325bc39-4cb3-49bb-b803-18c2ab4a24e7/dns-node-resolver/0.log" Jul 28 13:29:08.036700 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:08.036671 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vthpv_f2af6702-608a-497a-bcc5-d7eac4802553/node-ca/0.log" Jul 28 13:29:09.275804 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:09.275777 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jqjlx_b20ed721-ef7e-47eb-ad07-1f0a28163f89/discovery/0.log" Jul 28 13:29:10.370681 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:10.370626 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qfsc7_4a764e09-ea01-4cff-a9b1-0b14c995b676/serve-healthcheck-canary/0.log" Jul 28 13:29:10.939573 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:10.939547 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5hhgv/perf-node-gather-daemonset-bs724" Jul 28 13:29:10.972801 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:10.972772 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l7h6h_dba89490-b79c-47bb-b22c-2d4d923e5e5c/kube-rbac-proxy/0.log" Jul 28 13:29:10.994869 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:10.994844 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l7h6h_dba89490-b79c-47bb-b22c-2d4d923e5e5c/exporter/0.log" Jul 28 13:29:11.017634 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:11.017606 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l7h6h_dba89490-b79c-47bb-b22c-2d4d923e5e5c/extractor/0.log" Jul 28 13:29:14.481245 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:14.481212 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7974886bdb-72sjs_0dd05d93-2a5f-41c7-8996-e9560b576309/manager/0.log" Jul 28 13:29:14.498586 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:14.498562 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7974886bdb-72sjs_0dd05d93-2a5f-41c7-8996-e9560b576309/copy-manifests/0.log" Jul 28 13:29:15.636729 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:15.636695 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jqjlx_b20ed721-ef7e-47eb-ad07-1f0a28163f89/discovery/0.log" Jul 28 13:29:15.715812 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:15.715735 2573 scope.go:117] "RemoveContainer" containerID="81764b2721eb3a99d181b8979cc395ead87d6a36f5bf43c2438892c122c13c03" Jul 28 13:29:15.899051 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:15.899007 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-746578bc67-9lvhd_19d50472-8e5a-4a27-b39d-c3a6d632c8b1/manager/0.log" Jul 28 13:29:17.478649 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:17.478620 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jqjlx_b20ed721-ef7e-47eb-ad07-1f0a28163f89/discovery/0.log" Jul 28 13:29:21.091132 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:21.091108 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-kc66k_1b859f33-8b1d-4d81-b58f-da7a845c9d57/migrator/0.log" Jul 28 13:29:21.113125 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:21.113101 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-kc66k_1b859f33-8b1d-4d81-b58f-da7a845c9d57/graceful-termination/0.log" Jul 28 13:29:21.495650 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:21.495618 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4_971959e5-2ea2-4dde-8a74-43e61cfd6066/kube-storage-version-migrator-operator/1.log" Jul 28 13:29:21.497006 ip-10-0-136-145 kubenswrapper[2573]: I0728 13:29:21.496981 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-75wq4_971959e5-2ea2-4dde-8a74-43e61cfd6066/kube-storage-version-migrator-operator/0.log"