Jul 10 17:57:48.845131 ip-10-0-136-200 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 17:57:48.845142 ip-10-0-136-200 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 17:57:48.845151 ip-10-0-136-200 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 17:57:48.845507 ip-10-0-136-200 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 17:57:58.887891 ip-10-0-136-200 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 17:57:58.887912 ip-10-0-136-200 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 81990c8de99b4c83ab4c350bd09be2c5 -- Jul 10 17:59:54.882535 ip-10-0-136-200 systemd[1]: Starting Kubernetes Kubelet... Jul 10 17:59:55.316523 ip-10-0-136-200 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:59:55.316523 ip-10-0-136-200 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 17:59:55.316523 ip-10-0-136-200 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:59:55.316523 ip-10-0-136-200 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 17:59:55.316523 ip-10-0-136-200 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:59:55.317186 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.316578 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 17:59:55.321310 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321289 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:59:55.321310 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321307 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:59:55.321310 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321312 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:59:55.321310 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321316 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321320 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321325 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321329 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321334 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321338 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321341 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321345 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321348 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321352 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321363 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321367 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321371 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321374 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321379 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321382 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321386 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321390 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321394 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321398 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:59:55.321544 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321402 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321406 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321409 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321413 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321418 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321424 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321430 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321435 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321440 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321446 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321454 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321459 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321464 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321469 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321475 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321481 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321485 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321489 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321493 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:59:55.322374 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321498 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321502 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321506 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321510 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321514 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321519 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321550 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321556 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321561 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321565 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321570 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321574 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321578 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321582 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321587 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321591 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321595 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321599 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321604 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321607 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:59:55.323166 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321611 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321616 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321620 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321624 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321629 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321633 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321641 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321647 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321652 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321656 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321661 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321666 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321670 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321674 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321679 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321683 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321687 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321691 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321696 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:59:55.324009 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321700 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321704 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321708 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321712 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321716 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321721 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321725 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321729 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321734 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321738 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321742 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321746 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321750 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321755 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321759 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321763 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321767 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.321771 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.322956 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.322967 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:59:55.324841 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.322973 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.322978 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.322982 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.322987 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.322991 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.322996 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323000 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323004 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323008 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323013 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323017 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323021 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323025 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323029 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323033 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323037 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323043 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323050 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323054 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:59:55.325684 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323058 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323063 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323067 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323071 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323075 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323080 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323085 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323089 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323093 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323097 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323102 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323106 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323110 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323115 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323119 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323123 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323127 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323132 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323136 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323142 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:59:55.326389 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323146 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323150 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323155 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323159 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323162 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323167 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323170 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323174 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323178 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323183 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323187 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323191 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323195 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323199 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323206 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323229 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323235 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323244 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323248 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:59:55.327003 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323252 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323257 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323260 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323265 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323270 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323274 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323278 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323282 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323286 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323291 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323295 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323300 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323304 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323308 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323313 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323317 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323323 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323327 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323334 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323339 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:59:55.327618 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323344 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323349 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323354 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323359 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323363 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323368 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323373 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323377 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323382 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323386 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323392 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323396 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323400 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323404 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323409 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323413 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323417 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323421 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.323425 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324016 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:59:55.328121 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324026 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324030 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324037 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324044 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324049 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324054 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324059 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324063 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324067 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324071 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324075 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324080 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324084 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324087 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324092 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324096 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324100 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324104 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324108 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:59:55.328722 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324113 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324116 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324121 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324125 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324129 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324133 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324137 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324142 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324146 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324150 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324154 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324158 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324162 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324165 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324170 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324174 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324178 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324183 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324188 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:59:55.329184 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324192 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324195 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324201 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324205 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324209 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324231 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324235 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324239 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324243 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324247 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324251 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324255 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324260 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324264 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324268 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324272 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324277 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324281 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324285 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324290 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:59:55.329711 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324295 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324299 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324303 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324307 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324312 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324316 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324320 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324324 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324328 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324332 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324337 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324341 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324346 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324352 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324356 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324360 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324364 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324368 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324372 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324376 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:59:55.330181 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324383 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324388 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324393 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324397 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324402 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324407 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324412 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324416 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324421 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324425 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324429 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324434 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324438 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324442 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324446 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324451 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324455 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324459 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324463 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.324467 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324565 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 17:59:55.330698 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324575 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324588 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324595 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324602 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324607 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324614 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324621 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324625 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324630 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324635 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324640 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324646 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324651 2569 flags.go:64] FLAG: --cgroup-root="" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324655 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324660 2569 flags.go:64] FLAG: --client-ca-file="" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324664 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324669 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324675 2569 flags.go:64] FLAG: --cluster-domain="" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324680 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324685 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324691 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324696 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324705 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 17:59:55.331246 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324710 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324714 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324719 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324724 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324729 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324733 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324738 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324743 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324749 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324754 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324758 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324763 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324768 2569 flags.go:64] FLAG: --enable-server="true" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324773 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324779 2569 flags.go:64] FLAG: --event-burst="100" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324785 2569 flags.go:64] FLAG: --event-qps="50" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324790 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324794 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324799 2569 flags.go:64] FLAG: --eviction-hard="" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324805 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324810 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324814 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324819 2569 flags.go:64] FLAG: --eviction-soft="" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324823 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324828 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 17:59:55.331874 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324833 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324837 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324841 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324848 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324852 2569 flags.go:64] FLAG: --feature-gates="" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324858 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324864 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324869 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324874 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324879 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324884 2569 flags.go:64] FLAG: --help="false" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324888 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-136-200.ec2.internal" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324893 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324898 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324902 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324908 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324914 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324918 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324923 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324928 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324932 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324938 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324943 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324948 2569 flags.go:64] FLAG: --kube-reserved="" Jul 10 17:59:55.332515 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324952 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324957 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324962 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324966 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324971 2569 flags.go:64] FLAG: --lock-file="" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324975 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324980 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324985 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324993 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.324998 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325003 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325009 2569 flags.go:64] FLAG: --logging-format="text" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325014 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325019 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325025 2569 flags.go:64] FLAG: --manifest-url="" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325030 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325040 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325045 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325052 2569 flags.go:64] FLAG: --max-pods="110" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325056 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325061 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325066 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325070 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325076 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325081 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 17:59:55.333096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325086 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325098 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325103 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325107 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325112 2569 flags.go:64] FLAG: --pod-cidr="" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325118 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325127 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325132 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325136 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325141 2569 flags.go:64] FLAG: --port="10250" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325145 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325150 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c3d722360c7c3660" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325155 2569 flags.go:64] FLAG: --qos-reserved="" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325160 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325165 2569 flags.go:64] FLAG: --register-node="true" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325169 2569 flags.go:64] FLAG: --register-with-taints="" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325175 2569 flags.go:64] FLAG: --registry-burst="10" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325180 2569 flags.go:64] FLAG: --registry-qps="5" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325185 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325191 2569 flags.go:64] FLAG: --reserved-memory="" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325196 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325203 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325208 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325229 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325234 2569 flags.go:64] FLAG: --runonce="false" Jul 10 17:59:55.333725 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325238 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325243 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325248 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325253 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325258 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325263 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325268 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325273 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325277 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325282 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325287 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325291 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325296 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325302 2569 flags.go:64] FLAG: --system-cgroups="" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325307 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325316 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325320 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325324 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325331 2569 flags.go:64] FLAG: --tls-min-version="" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325336 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325341 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325346 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325350 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325355 2569 flags.go:64] FLAG: --v="2" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325361 2569 flags.go:64] FLAG: --version="false" Jul 10 17:59:55.334399 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325367 2569 flags.go:64] FLAG: --vmodule="" Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325375 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.325381 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325560 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325567 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325572 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325577 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325582 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325587 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325591 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325596 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325600 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325605 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325610 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325614 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325619 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325623 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325627 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325632 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325636 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:59:55.335036 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325640 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325645 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325649 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325653 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325657 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325662 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325666 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325670 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325674 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325679 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325682 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325686 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325690 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325697 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325701 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325707 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325711 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325716 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325720 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325724 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:59:55.335559 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325728 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325733 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325737 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325742 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325746 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325750 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325754 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325758 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325762 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325766 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325771 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325775 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325782 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325787 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325794 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325799 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325804 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325808 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325813 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:59:55.336056 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325818 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325823 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325827 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325833 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325839 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325844 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325850 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325855 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325861 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325865 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325870 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325875 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325879 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325883 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325887 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325891 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325895 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325899 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325904 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:59:55.336688 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325908 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325912 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325916 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325921 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325925 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325929 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325933 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325937 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325941 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325946 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325951 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325956 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325960 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325964 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325968 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325973 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325979 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325985 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325989 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.325996 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:59:55.337507 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.326001 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.326007 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.326012 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.326016 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.326024 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.333378 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.333395 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.334242 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.334251 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334316 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334321 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334324 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334327 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334330 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334333 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:59:55.338052 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334336 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334339 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334341 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334344 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334347 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334350 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334353 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334355 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334358 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334360 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334363 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334365 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334368 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334370 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334373 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334376 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334378 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334382 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334386 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:59:55.338472 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334389 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334392 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334395 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334397 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334407 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334410 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334413 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334416 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334419 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334421 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334424 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334426 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334429 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334432 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334434 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334436 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334439 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334442 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334444 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334447 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:59:55.338925 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334449 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334452 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334454 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334457 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334460 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334463 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334465 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334468 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334471 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334474 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334476 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334478 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334481 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334483 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334486 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334488 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334491 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334500 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334503 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334505 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:59:55.339517 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334508 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334510 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334513 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334515 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334518 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334521 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334523 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334526 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334528 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334531 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334533 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334536 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334538 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334540 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334543 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334545 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334548 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334550 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334553 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334555 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:59:55.340000 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334558 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334560 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334563 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334565 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334568 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334572 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334576 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334580 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334590 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334594 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334605 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334608 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334611 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334614 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:59:55.340510 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.334619 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334787 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334792 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334795 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334798 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334801 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334804 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334806 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334809 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334811 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334814 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334817 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334819 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334821 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334824 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334826 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334829 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334831 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334833 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334836 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:59:55.340920 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334839 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334842 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334844 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334847 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334851 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334855 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334859 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334862 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334865 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334873 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334876 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334879 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334881 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334884 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334886 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334889 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334892 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334894 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334897 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:59:56.781034 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334899 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334902 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334904 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334907 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334909 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334911 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334914 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334916 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334919 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334922 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334924 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334926 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334930 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334932 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334935 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334937 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334940 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334942 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334944 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334947 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:59:56.781774 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334950 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334952 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334955 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334963 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334966 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334968 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334971 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334973 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334975 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334978 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334980 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334983 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334985 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334988 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334990 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334993 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334995 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.334998 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335000 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335003 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:59:56.782440 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335005 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335008 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335010 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335012 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335015 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335018 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335020 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335023 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335025 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335028 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335030 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335033 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335036 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335040 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335043 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335046 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335055 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335058 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335060 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335063 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:59:56.782980 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:55.335065 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:59:56.783472 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.335070 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:59:56.783472 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.335203 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 17:59:56.783472 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.339193 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 17:59:56.783472 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.340634 2569 server.go:1019] "Starting client certificate rotation" Jul 10 17:59:56.783472 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.340748 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 17:59:56.783472 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:55.340786 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 17:59:56.783472 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.782804 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 17:59:56.783985 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.783968 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 17:59:56.797085 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.797065 2569 log.go:25] "Validated CRI v1 runtime API" Jul 10 17:59:56.799141 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.799127 2569 log.go:25] "Validated CRI v1 image API" Jul 10 17:59:56.799941 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.799927 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 17:59:56.802294 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.802272 2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 96e8f6f6-77db-434c-ac06-22228487b176:/dev/nvme0n1p3 c8e92331-8196-4de1-b77a-5c80547efb6f:/dev/nvme0n1p4] Jul 10 17:59:56.802367 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.802292 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 17:59:56.807071 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.806955 2569 manager.go:216] Machine: {Timestamp:2026-07-10 17:59:56.806344092 +0000 UTC m=+0.352807224 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101302 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f75b1a47e45f9a74711bb9bedc531 SystemUUID:ec2f75b1-a47e-45f9-a747-11bb9bedc531 BootID:81990c8d-e99b-4c83-ab4c-350bd09be2c5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e5:d1:f0:58:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e5:d1:f0:58:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:19:69:96:bf:b8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 17:59:56.807071 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.807057 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 17:59:56.807216 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.807159 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 17:59:56.808271 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.808255 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 17:59:56.811693 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.811665 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 17:59:56.812360 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.811698 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 17:59:56.813760 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.813541 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 17:59:56.813859 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.813767 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 17:59:56.813859 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.813777 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 17:59:56.814745 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.814733 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 17:59:56.815656 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.815646 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 17:59:56.819168 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.819157 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 10 17:59:56.819304 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.819288 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 17:59:56.819337 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.819323 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 10 17:59:56.819364 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.819335 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 17:59:56.819364 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.819356 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 17:59:56.819414 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.819367 2569 kubelet.go:398] "Adding apiserver pod source" Jul 10 17:59:56.819414 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.819377 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 17:59:56.819722 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.819712 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 17:59:56.819757 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.819730 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 17:59:56.821841 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.821819 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 17:59:56.823720 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.823706 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 17:59:56.823774 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.823737 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 17:59:56.824386 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.824374 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 17:59:56.824434 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.824391 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 17:59:56.824434 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.824397 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 17:59:56.824434 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.824402 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 17:59:56.824434 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.824410 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 17:59:56.824434 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.824415 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 17:59:56.824561 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.824503 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 17:59:56.824561 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.824509 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 17:59:56.824561 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.824515 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 17:59:56.824561 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.824521 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 17:59:56.824561 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.824529 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 17:59:56.824561 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.824537 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 17:59:56.826088 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.826076 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 17:59:56.826122 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.826091 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 17:59:56.829508 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.829492 2569 server.go:1268] "Started kubelet" Jul 10 17:59:56.829642 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.829588 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 17:59:56.829842 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.829802 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 17:59:56.830161 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:56.830088 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 17:59:56.830161 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.830154 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 17:59:56.830354 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:56.830249 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 17:59:56.830354 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.830311 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-200.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 17:59:56.830741 ip-10-0-136-200 systemd[1]: Started Kubernetes Kubelet. Jul 10 17:59:56.831394 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.831347 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 17:59:56.831784 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.831772 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 10 17:59:56.832639 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.832621 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4ccbt" Jul 10 17:59:56.835721 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.835706 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 17:59:56.835798 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.835743 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 17:59:56.835892 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:56.835872 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-200.ec2.internal\" not found" Jul 10 17:59:56.835947 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:56.835916 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 17:59:56.836337 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.836316 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 17:59:56.836337 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.836316 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 17:59:56.836337 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.836344 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 17:59:56.836533 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.836430 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 10 17:59:56.836533 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.836438 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 10 17:59:56.836740 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.836722 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 17:59:56.836740 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.836739 2569 factory.go:55] Registering systemd factory Jul 10 17:59:56.836878 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.836748 2569 factory.go:223] Registration of the systemd container factory successfully Jul 10 17:59:56.836987 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.836972 2569 factory.go:153] Registering CRI-O factory Jul 10 17:59:56.836987 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.836986 2569 factory.go:223] Registration of the crio container factory successfully Jul 10 17:59:56.837096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.837003 2569 factory.go:103] Registering Raw factory Jul 10 17:59:56.837096 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.837012 2569 manager.go:1196] Started watching for new ooms in manager Jul 10 17:59:56.837736 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.837722 2569 manager.go:349] Starting recovery of all containers Jul 10 17:59:56.840297 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.840272 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4ccbt" Jul 10 17:59:56.840508 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:56.840469 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 17:59:56.840711 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:56.840687 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 17:59:56.841875 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:56.840554 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-200.ec2.internal.18c0ff84769ca87a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-200.ec2.internal,UID:ip-10-0-136-200.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-200.ec2.internal,},FirstTimestamp:2026-07-10 17:59:56.829460602 +0000 UTC m=+0.375923734,LastTimestamp:2026-07-10 17:59:56.829460602 +0000 UTC m=+0.375923734,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-200.ec2.internal,}" Jul 10 17:59:56.850731 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.850582 2569 manager.go:354] Recovery completed Jul 10 17:59:56.854882 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.854868 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:59:56.856594 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.856577 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:59:56.856654 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.856602 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:59:56.856654 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.856616 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:59:56.857147 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.857130 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 17:59:56.857147 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.857140 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:59:56.857147 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.857145 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 17:59:56.857294 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.857162 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 10 17:59:56.857750 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.857738 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:59:56.857848 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.857765 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:59:56.857848 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.857780 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:59:56.858726 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.858713 2569 policy_none.go:49] "None policy: Start" Jul 10 17:59:56.858788 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.858730 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 17:59:56.858788 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:56.858688 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-200.ec2.internal.18c0ff84783aa87e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-200.ec2.internal,UID:ip-10-0-136-200.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-200.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-200.ec2.internal,},FirstTimestamp:2026-07-10 17:59:56.85659251 +0000 UTC m=+0.403055643,LastTimestamp:2026-07-10 17:59:56.85659251 +0000 UTC m=+0.403055643,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-200.ec2.internal,}" Jul 10 17:59:56.858788 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.858743 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 17:59:56.860000 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.859988 2569 policy_none.go:47] "Start" Jul 10 17:59:56.894359 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.894345 2569 manager.go:337] "Starting Device Plugin manager" Jul 10 17:59:56.894433 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:56.894372 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 17:59:56.894433 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.894384 2569 server.go:87] "Starting device plugin registration server" Jul 10 17:59:56.894616 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.894601 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 17:59:56.894702 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.894614 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 17:59:56.894754 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.894715 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 17:59:56.894843 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.894794 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 17:59:56.894843 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.894830 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 17:59:56.895943 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:56.895924 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 17:59:56.896002 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:56.895969 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-200.ec2.internal\" not found" Jul 10 17:59:56.976601 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.976572 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 17:59:56.977734 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.977713 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 17:59:56.977804 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.977747 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 17:59:56.977804 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.977772 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 17:59:56.977910 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:56.977827 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 17:59:56.980041 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.980023 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:59:56.995036 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.995018 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:59:56.995748 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.995731 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:59:56.995842 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.995762 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:59:56.995842 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.995775 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:59:56.995842 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:56.995797 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.003734 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.003720 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.003837 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:57.003742 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-200.ec2.internal\": node \"ip-10-0-136-200.ec2.internal\" not found" Jul 10 17:59:57.034871 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:57.034804 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-200.ec2.internal\" not found" Jul 10 17:59:57.078891 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.078856 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-200.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal"] Jul 10 17:59:57.078972 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.078943 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:59:57.080478 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.080462 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:59:57.080547 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.080493 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:59:57.080547 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.080503 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:59:57.081637 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.081625 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:59:57.081756 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.081736 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.081825 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.081773 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:59:57.082341 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.082326 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:59:57.082395 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.082346 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:59:57.082395 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.082356 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:59:57.082943 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.082930 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:59:57.082992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.082953 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:59:57.082992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.082968 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:59:57.083486 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.083472 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.083549 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.083496 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:59:57.084116 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.084100 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:59:57.084210 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.084127 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:59:57.084210 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.084141 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:59:57.122400 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:57.122380 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-200.ec2.internal\" not found" node="ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.127219 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:57.127202 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-200.ec2.internal\" not found" node="ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.135684 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:57.135669 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-200.ec2.internal\" not found" Jul 10 17:59:57.139014 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.138986 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f403d9863d495b5a87ed67af63e35db-config\") pod \"kube-apiserver-proxy-ip-10-0-136-200.ec2.internal\" (UID: \"4f403d9863d495b5a87ed67af63e35db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.139099 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.139021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4643467acf069073b49fa3fc327d952b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal\" (UID: \"4643467acf069073b49fa3fc327d952b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.139099 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.139047 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4643467acf069073b49fa3fc327d952b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal\" (UID: \"4643467acf069073b49fa3fc327d952b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.236110 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:57.236078 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-200.ec2.internal\" not found" Jul 10 17:59:57.239323 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.239308 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f403d9863d495b5a87ed67af63e35db-config\") pod \"kube-apiserver-proxy-ip-10-0-136-200.ec2.internal\" (UID: \"4f403d9863d495b5a87ed67af63e35db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.239380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.239331 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4643467acf069073b49fa3fc327d952b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal\" (UID: \"4643467acf069073b49fa3fc327d952b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.239380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.239348 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4643467acf069073b49fa3fc327d952b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal\" (UID: \"4643467acf069073b49fa3fc327d952b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.239473 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.239374 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4643467acf069073b49fa3fc327d952b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal\" (UID: \"4643467acf069073b49fa3fc327d952b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.239473 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.239404 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4643467acf069073b49fa3fc327d952b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal\" (UID: \"4643467acf069073b49fa3fc327d952b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.239473 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.239409 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f403d9863d495b5a87ed67af63e35db-config\") pod \"kube-apiserver-proxy-ip-10-0-136-200.ec2.internal\" (UID: \"4f403d9863d495b5a87ed67af63e35db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.336761 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:57.336695 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-200.ec2.internal\" not found" Jul 10 17:59:57.425101 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.425072 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.430854 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.430837 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.437418 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:57.437398 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-200.ec2.internal\" not found" Jul 10 17:59:57.537883 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:57.537846 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-200.ec2.internal\" not found" Jul 10 17:59:57.638398 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:57.638334 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-200.ec2.internal\" not found" Jul 10 17:59:57.664051 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.664028 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:59:57.738423 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:57.738395 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-200.ec2.internal\" not found" Jul 10 17:59:57.779693 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.779672 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 17:59:57.780237 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.779787 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 17:59:57.780237 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.779839 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 17:59:57.807978 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.807956 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:59:57.838473 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:57.838447 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-200.ec2.internal\" not found" Jul 10 17:59:57.847737 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.847707 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 17:54:56 +0000 UTC" deadline="2028-04-08 00:31:39.78768967 +0000 UTC" Jul 10 17:59:57.847737 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.847731 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15294h31m41.939960653s" Jul 10 17:59:57.904279 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.904205 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:59:57.936095 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.936070 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.940294 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:57.940265 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4643467acf069073b49fa3fc327d952b.slice/crio-50511f03d05802aa9074d5f54fec72bb0fc41bbdc8e602db803430337716d604 WatchSource:0}: Error finding container 50511f03d05802aa9074d5f54fec72bb0fc41bbdc8e602db803430337716d604: Status 404 returned error can't find the container with id 50511f03d05802aa9074d5f54fec72bb0fc41bbdc8e602db803430337716d604 Jul 10 17:59:57.940911 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:57.940891 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f403d9863d495b5a87ed67af63e35db.slice/crio-af818f3fc2c5b58f5f3b73b00f33bf393be2a9b090d11381d4191943ecbdf347 WatchSource:0}: Error finding container af818f3fc2c5b58f5f3b73b00f33bf393be2a9b090d11381d4191943ecbdf347: Status 404 returned error can't find the container with id af818f3fc2c5b58f5f3b73b00f33bf393be2a9b090d11381d4191943ecbdf347 Jul 10 17:59:57.943694 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.943670 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 17:59:57.944682 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.944643 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal" Jul 10 17:59:57.944945 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.944930 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:59:57.954088 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.954075 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 17:59:57.981067 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.981021 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal" event={"ID":"4643467acf069073b49fa3fc327d952b","Type":"ContainerStarted","Data":"50511f03d05802aa9074d5f54fec72bb0fc41bbdc8e602db803430337716d604"} Jul 10 17:59:57.982071 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:57.982051 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-200.ec2.internal" event={"ID":"4f403d9863d495b5a87ed67af63e35db","Type":"ContainerStarted","Data":"af818f3fc2c5b58f5f3b73b00f33bf393be2a9b090d11381d4191943ecbdf347"} Jul 10 17:59:58.819984 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.819955 2569 apiserver.go:52] "Watching apiserver" Jul 10 17:59:58.827987 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.827603 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 17:59:58.828206 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.828176 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-z5g5q","openshift-network-diagnostics/network-check-target-z7dtl","openshift-network-operator/iptables-alerter-4pm4x","openshift-ovn-kubernetes/ovnkube-node-6z78q","kube-system/global-pull-secret-syncer-g8xxt","kube-system/konnectivity-agent-lbc2g","kube-system/kube-apiserver-proxy-ip-10-0-136-200.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4","openshift-cluster-node-tuning-operator/tuned-5p5nk","openshift-dns/node-resolver-g7nkb","openshift-image-registry/node-ca-f4xrb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal","openshift-multus/multus-additional-cni-plugins-dmqqt","openshift-multus/multus-qp8r9"] Jul 10 17:59:58.836684 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.836659 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 17:59:58.837502 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.837199 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 17:59:58.837502 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:58.837283 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 17:59:58.838596 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.838488 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 17:59:58.838596 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:58.838562 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 17:59:58.839736 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.839712 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4pm4x" Jul 10 17:59:58.841451 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.841048 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.841893 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.841874 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-52qbr\"" Jul 10 17:59:58.842722 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.842300 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 17:59:58.842722 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.842452 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:59:58.842722 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.842558 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 17:59:58.842722 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.842649 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 17:59:58.842722 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:58.842715 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 17:59:58.843374 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.843126 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 17:59:58.844846 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.844156 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 17:59:58.844846 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.844213 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 17:59:58.844846 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.844392 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lbc2g" Jul 10 17:59:58.844846 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.844451 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 17:59:58.846172 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.845800 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:58.847094 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.846967 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 17:59:58.848763 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.847199 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 17:59:58.848763 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.847378 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs\") pod \"network-metrics-daemon-z5g5q\" (UID: \"c1121149-7010-4fb2-a378-dc79d124db58\") " pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 17:59:58.848763 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.847412 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w87kt\" (UniqueName: \"kubernetes.io/projected/c1121149-7010-4fb2-a378-dc79d124db58-kube-api-access-w87kt\") pod \"network-metrics-daemon-z5g5q\" (UID: \"c1121149-7010-4fb2-a378-dc79d124db58\") " pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 17:59:58.848763 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.847513 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.848763 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.847606 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 17:59:58.848763 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.848035 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 17:59:58.848763 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.848500 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sm85d\"" Jul 10 17:59:58.848763 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.848687 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-d4c4v\"" Jul 10 17:59:58.849203 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.848914 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 17:59:58.849464 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.849447 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 17:59:58.849656 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.849641 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 17:59:58.850025 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.850006 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jxlnc\"" Jul 10 17:59:58.850134 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.850107 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:59:58.850309 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.850290 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7f87z\"" Jul 10 17:59:58.850888 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.850775 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 17:59:58.851662 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.851601 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g7nkb" Jul 10 17:59:58.854311 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.853854 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 17:59:58.854311 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.854075 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6jspl\"" Jul 10 17:59:58.854461 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.854414 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 17:59:58.854517 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.854498 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 17:59:58.855527 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.854694 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f4xrb" Jul 10 17:59:58.856875 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.856509 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:58.857056 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.857037 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 17:59:58.857145 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.857127 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 17:59:58.857255 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.857232 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bmrl7\"" Jul 10 17:59:58.857534 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.857474 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 17:59:58.858481 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.858462 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.859115 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.858973 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 17:59:58.859115 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.859001 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 17:59:58.859115 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.859052 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 17:59:58.859326 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.859139 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dkx8d\"" Jul 10 17:59:58.859326 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.859255 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 17:59:58.859569 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.859553 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 17:59:58.860458 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.860439 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 17:59:58.860926 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.860909 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-n6xqx\"" Jul 10 17:59:58.885906 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.885876 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m6g8m" Jul 10 17:59:58.888018 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.887995 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:59:58.903203 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.903123 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m6g8m" Jul 10 17:59:58.937489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.937460 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 17:59:58.947580 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947559 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-multus-socket-dir-parent\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.947704 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947585 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-sysctl-conf\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.947704 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947612 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2890ff6-6c54-44b9-94b4-590301075523-host-slash\") pod \"iptables-alerter-4pm4x\" (UID: \"c2890ff6-6c54-44b9-94b4-590301075523\") " pod="openshift-network-operator/iptables-alerter-4pm4x" Jul 10 17:59:58.947704 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947637 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-run-netns\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.947704 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947659 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-run-ovn\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.947704 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947680 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-cni-bin\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.947992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947710 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-cni-netd\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.947992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947729 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-os-release\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.947992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947748 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-device-dir\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:58.947992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947774 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqp92\" (UniqueName: \"kubernetes.io/projected/3183f94d-897c-4803-af1c-b40745a1268d-kube-api-access-kqp92\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:58.947992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947798 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456cd2fa-e3a0-40f3-b865-15d505e82bbe-tmp-dir\") pod \"node-resolver-g7nkb\" (UID: \"456cd2fa-e3a0-40f3-b865-15d505e82bbe\") " pod="openshift-dns/node-resolver-g7nkb" Jul 10 17:59:58.947992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947836 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c2890ff6-6c54-44b9-94b4-590301075523-iptables-alerter-script\") pod \"iptables-alerter-4pm4x\" (UID: \"c2890ff6-6c54-44b9-94b4-590301075523\") " pod="openshift-network-operator/iptables-alerter-4pm4x" Jul 10 17:59:58.947992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947861 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/159d239e-3d57-4384-95cd-e37f9bdcc401-env-overrides\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.947992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947882 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-kubelet-config\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 17:59:58.947992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947912 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:58.947992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947942 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-kubernetes\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.947992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947963 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-tuned\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.947992 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.947985 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-tmp\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948006 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/456cd2fa-e3a0-40f3-b865-15d505e82bbe-hosts-file\") pod \"node-resolver-g7nkb\" (UID: \"456cd2fa-e3a0-40f3-b865-15d505e82bbe\") " pod="openshift-dns/node-resolver-g7nkb" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhqk\" (UniqueName: \"kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk\") pod \"network-check-target-z7dtl\" (UID: \"9766e246-9e80-449e-b500-e290fa4ac5c1\") " pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948046 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27v82\" (UniqueName: \"kubernetes.io/projected/e7d64404-7c29-458a-bc00-2a745b9cbc85-kube-api-access-27v82\") pod \"node-ca-f4xrb\" (UID: \"e7d64404-7c29-458a-bc00-2a745b9cbc85\") " pod="openshift-image-registry/node-ca-f4xrb" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948070 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-var-lib-cni-bin\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948091 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-lib-modules\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948114 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdhzs\" (UniqueName: \"kubernetes.io/projected/c2890ff6-6c54-44b9-94b4-590301075523-kube-api-access-tdhzs\") pod \"iptables-alerter-4pm4x\" (UID: \"c2890ff6-6c54-44b9-94b4-590301075523\") " pod="openshift-network-operator/iptables-alerter-4pm4x" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948136 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ts8\" (UniqueName: \"kubernetes.io/projected/159d239e-3d57-4384-95cd-e37f9bdcc401-kube-api-access-56ts8\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948161 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/499fd8cb-4e54-4820-a90d-c55159a3910a-os-release\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948181 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-hostroot\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948220 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-dbus\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948272 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-socket-dir\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948297 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cz4j\" (UniqueName: \"kubernetes.io/projected/456cd2fa-e3a0-40f3-b865-15d505e82bbe-kube-api-access-7cz4j\") pod \"node-resolver-g7nkb\" (UID: \"456cd2fa-e3a0-40f3-b865-15d505e82bbe\") " pod="openshift-dns/node-resolver-g7nkb" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948320 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-etc-openvswitch\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948345 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-run-openvswitch\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948390 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/499fd8cb-4e54-4820-a90d-c55159a3910a-system-cni-dir\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:58.948449 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948421 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/499fd8cb-4e54-4820-a90d-c55159a3910a-cnibin\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948462 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-run-netns\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948492 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-run-multus-certs\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948536 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-run\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948571 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-node-log\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948615 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-run-ovn-kubernetes\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948652 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/159d239e-3d57-4384-95cd-e37f9bdcc401-ovnkube-script-lib\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948679 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-system-cni-dir\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948703 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-var-lib-kubelet\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948727 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb493400-515b-4ed8-934f-038f1aae45ed-agent-certs\") pod \"konnectivity-agent-lbc2g\" (UID: \"fb493400-515b-4ed8-934f-038f1aae45ed\") " pod="kube-system/konnectivity-agent-lbc2g" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948750 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-multus-conf-dir\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948774 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-systemd\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948837 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-host\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948861 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4frw6\" (UniqueName: \"kubernetes.io/projected/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-kube-api-access-4frw6\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948889 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-run-systemd\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948921 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-var-lib-openvswitch\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.949146 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948954 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/499fd8cb-4e54-4820-a90d-c55159a3910a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.948978 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8dkz\" (UniqueName: \"kubernetes.io/projected/499fd8cb-4e54-4820-a90d-c55159a3910a-kube-api-access-d8dkz\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949009 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-etc-kubernetes\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949032 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949058 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/499fd8cb-4e54-4820-a90d-c55159a3910a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949081 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-var-lib-cni-multus\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d2dd6c1-df2f-4087-b03f-c18d9418c332-multus-daemon-config\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949140 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-sysconfig\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949161 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7d64404-7c29-458a-bc00-2a745b9cbc85-serviceca\") pod \"node-ca-f4xrb\" (UID: \"e7d64404-7c29-458a-bc00-2a745b9cbc85\") " pod="openshift-image-registry/node-ca-f4xrb" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949183 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5bcb\" (UniqueName: \"kubernetes.io/projected/2d2dd6c1-df2f-4087-b03f-c18d9418c332-kube-api-access-p5bcb\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949205 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-slash\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949227 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-log-socket\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949247 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7d64404-7c29-458a-bc00-2a745b9cbc85-host\") pod \"node-ca-f4xrb\" (UID: \"e7d64404-7c29-458a-bc00-2a745b9cbc85\") " pod="openshift-image-registry/node-ca-f4xrb" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949270 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/499fd8cb-4e54-4820-a90d-c55159a3910a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949305 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-run-k8s-cni-cncf-io\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949339 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb493400-515b-4ed8-934f-038f1aae45ed-konnectivity-ca\") pod \"konnectivity-agent-lbc2g\" (UID: \"fb493400-515b-4ed8-934f-038f1aae45ed\") " pod="kube-system/konnectivity-agent-lbc2g" Jul 10 17:59:58.949738 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949375 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs\") pod \"network-metrics-daemon-z5g5q\" (UID: \"c1121149-7010-4fb2-a378-dc79d124db58\") " pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949403 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-multus-cni-dir\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949418 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949435 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-sysctl-d\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949455 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-var-lib-kubelet\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949478 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/499fd8cb-4e54-4820-a90d-c55159a3910a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:58.949494 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:58.949575 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs podName:c1121149-7010-4fb2-a378-dc79d124db58 nodeName:}" failed. No retries permitted until 2026-07-10 17:59:59.449545273 +0000 UTC m=+2.996008409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs") pod "network-metrics-daemon-z5g5q" (UID: "c1121149-7010-4fb2-a378-dc79d124db58") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949497 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-sys\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949609 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-kubelet\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949630 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/159d239e-3d57-4384-95cd-e37f9bdcc401-ovn-node-metrics-cert\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w87kt\" (UniqueName: \"kubernetes.io/projected/c1121149-7010-4fb2-a378-dc79d124db58-kube-api-access-w87kt\") pod \"network-metrics-daemon-z5g5q\" (UID: \"c1121149-7010-4fb2-a378-dc79d124db58\") " pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949665 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-cnibin\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949685 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-sys-fs\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949715 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-systemd-units\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949733 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/159d239e-3d57-4384-95cd-e37f9bdcc401-ovnkube-config\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:58.950368 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949747 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-registration-dir\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:58.950945 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949761 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-modprobe-d\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:58.950945 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.949783 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d2dd6c1-df2f-4087-b03f-c18d9418c332-cni-binary-copy\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:58.956709 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.956686 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 17:59:58.960476 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:58.960438 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w87kt\" (UniqueName: \"kubernetes.io/projected/c1121149-7010-4fb2-a378-dc79d124db58-kube-api-access-w87kt\") pod \"network-metrics-daemon-z5g5q\" (UID: \"c1121149-7010-4fb2-a378-dc79d124db58\") " pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 17:59:59.050893 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.050852 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/499fd8cb-4e54-4820-a90d-c55159a3910a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.051056 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.050900 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-var-lib-cni-multus\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.051056 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.050924 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d2dd6c1-df2f-4087-b03f-c18d9418c332-multus-daemon-config\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.051056 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.050945 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-sysconfig\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.051056 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.050967 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7d64404-7c29-458a-bc00-2a745b9cbc85-serviceca\") pod \"node-ca-f4xrb\" (UID: \"e7d64404-7c29-458a-bc00-2a745b9cbc85\") " pod="openshift-image-registry/node-ca-f4xrb" Jul 10 17:59:59.051056 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.050987 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5bcb\" (UniqueName: \"kubernetes.io/projected/2d2dd6c1-df2f-4087-b03f-c18d9418c332-kube-api-access-p5bcb\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.051056 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051008 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-slash\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.051056 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051029 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-log-socket\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.051056 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051033 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-sysconfig\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.051056 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.050984 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-var-lib-cni-multus\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051049 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7d64404-7c29-458a-bc00-2a745b9cbc85-host\") pod \"node-ca-f4xrb\" (UID: \"e7d64404-7c29-458a-bc00-2a745b9cbc85\") " pod="openshift-image-registry/node-ca-f4xrb" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051092 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-log-socket\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051105 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/499fd8cb-4e54-4820-a90d-c55159a3910a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051130 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7d64404-7c29-458a-bc00-2a745b9cbc85-host\") pod \"node-ca-f4xrb\" (UID: \"e7d64404-7c29-458a-bc00-2a745b9cbc85\") " pod="openshift-image-registry/node-ca-f4xrb" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051138 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-run-k8s-cni-cncf-io\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051165 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-slash\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051167 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb493400-515b-4ed8-934f-038f1aae45ed-konnectivity-ca\") pod \"konnectivity-agent-lbc2g\" (UID: \"fb493400-515b-4ed8-934f-038f1aae45ed\") " pod="kube-system/konnectivity-agent-lbc2g" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051306 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-multus-cni-dir\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051335 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051361 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-sysctl-d\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051381 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-multus-cni-dir\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051384 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-var-lib-kubelet\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051427 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/499fd8cb-4e54-4820-a90d-c55159a3910a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051432 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-var-lib-kubelet\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051454 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-sys\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051479 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-kubelet\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.051489 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051487 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051521 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d2dd6c1-df2f-4087-b03f-c18d9418c332-multus-daemon-config\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051538 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-sys\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051536 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/499fd8cb-4e54-4820-a90d-c55159a3910a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051528 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-kubelet\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051553 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/159d239e-3d57-4384-95cd-e37f9bdcc401-ovn-node-metrics-cert\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051570 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-sysctl-d\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051606 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-cnibin\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051610 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-run-k8s-cni-cncf-io\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051631 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-sys-fs\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051656 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-systemd-units\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051666 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-cnibin\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051686 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/159d239e-3d57-4384-95cd-e37f9bdcc401-ovnkube-config\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051711 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-registration-dir\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051713 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/499fd8cb-4e54-4820-a90d-c55159a3910a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051719 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-sys-fs\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051736 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-modprobe-d\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051745 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-systemd-units\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.052303 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d2dd6c1-df2f-4087-b03f-c18d9418c332-cni-binary-copy\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051783 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-registration-dir\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051785 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-multus-socket-dir-parent\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051841 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-sysctl-conf\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051853 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-multus-socket-dir-parent\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051866 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2890ff6-6c54-44b9-94b4-590301075523-host-slash\") pod \"iptables-alerter-4pm4x\" (UID: \"c2890ff6-6c54-44b9-94b4-590301075523\") " pod="openshift-network-operator/iptables-alerter-4pm4x" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051869 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-modprobe-d\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051906 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2890ff6-6c54-44b9-94b4-590301075523-host-slash\") pod \"iptables-alerter-4pm4x\" (UID: \"c2890ff6-6c54-44b9-94b4-590301075523\") " pod="openshift-network-operator/iptables-alerter-4pm4x" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051905 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-run-netns\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051935 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-run-ovn\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051937 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-run-netns\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051971 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-cni-bin\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051981 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-run-ovn\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.051997 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-cni-netd\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052021 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-os-release\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052040 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/499fd8cb-4e54-4820-a90d-c55159a3910a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052065 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-device-dir\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.053154 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052097 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqp92\" (UniqueName: \"kubernetes.io/projected/3183f94d-897c-4803-af1c-b40745a1268d-kube-api-access-kqp92\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052116 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-sysctl-conf\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052121 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456cd2fa-e3a0-40f3-b865-15d505e82bbe-tmp-dir\") pod \"node-resolver-g7nkb\" (UID: \"456cd2fa-e3a0-40f3-b865-15d505e82bbe\") " pod="openshift-dns/node-resolver-g7nkb" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052132 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-device-dir\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052146 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c2890ff6-6c54-44b9-94b4-590301075523-iptables-alerter-script\") pod \"iptables-alerter-4pm4x\" (UID: \"c2890ff6-6c54-44b9-94b4-590301075523\") " pod="openshift-network-operator/iptables-alerter-4pm4x" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052153 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-cni-bin\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/159d239e-3d57-4384-95cd-e37f9bdcc401-env-overrides\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052185 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-os-release\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052195 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-kubelet-config\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052265 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-kubernetes\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-tuned\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052307 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-tmp\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052330 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/456cd2fa-e3a0-40f3-b865-15d505e82bbe-hosts-file\") pod \"node-resolver-g7nkb\" (UID: \"456cd2fa-e3a0-40f3-b865-15d505e82bbe\") " pod="openshift-dns/node-resolver-g7nkb" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052365 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb493400-515b-4ed8-934f-038f1aae45ed-konnectivity-ca\") pod \"konnectivity-agent-lbc2g\" (UID: \"fb493400-515b-4ed8-934f-038f1aae45ed\") " pod="kube-system/konnectivity-agent-lbc2g" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052353 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhqk\" (UniqueName: \"kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk\") pod \"network-check-target-z7dtl\" (UID: \"9766e246-9e80-449e-b500-e290fa4ac5c1\") " pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052391 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d2dd6c1-df2f-4087-b03f-c18d9418c332-cni-binary-copy\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052511 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7d64404-7c29-458a-bc00-2a745b9cbc85-serviceca\") pod \"node-ca-f4xrb\" (UID: \"e7d64404-7c29-458a-bc00-2a745b9cbc85\") " pod="openshift-image-registry/node-ca-f4xrb" Jul 10 17:59:59.053981 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052561 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/456cd2fa-e3a0-40f3-b865-15d505e82bbe-tmp-dir\") pod \"node-resolver-g7nkb\" (UID: \"456cd2fa-e3a0-40f3-b865-15d505e82bbe\") " pod="openshift-dns/node-resolver-g7nkb" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052569 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/159d239e-3d57-4384-95cd-e37f9bdcc401-env-overrides\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27v82\" (UniqueName: \"kubernetes.io/projected/e7d64404-7c29-458a-bc00-2a745b9cbc85-kube-api-access-27v82\") pod \"node-ca-f4xrb\" (UID: \"e7d64404-7c29-458a-bc00-2a745b9cbc85\") " pod="openshift-image-registry/node-ca-f4xrb" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052642 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-cni-netd\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052646 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-kubelet-config\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052655 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-var-lib-cni-bin\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052690 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-lib-modules\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052696 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-kubernetes\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052699 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c2890ff6-6c54-44b9-94b4-590301075523-iptables-alerter-script\") pod \"iptables-alerter-4pm4x\" (UID: \"c2890ff6-6c54-44b9-94b4-590301075523\") " pod="openshift-network-operator/iptables-alerter-4pm4x" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052713 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdhzs\" (UniqueName: \"kubernetes.io/projected/c2890ff6-6c54-44b9-94b4-590301075523-kube-api-access-tdhzs\") pod \"iptables-alerter-4pm4x\" (UID: \"c2890ff6-6c54-44b9-94b4-590301075523\") " pod="openshift-network-operator/iptables-alerter-4pm4x" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ts8\" (UniqueName: \"kubernetes.io/projected/159d239e-3d57-4384-95cd-e37f9bdcc401-kube-api-access-56ts8\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052768 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/499fd8cb-4e54-4820-a90d-c55159a3910a-os-release\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052787 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052791 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/456cd2fa-e3a0-40f3-b865-15d505e82bbe-hosts-file\") pod \"node-resolver-g7nkb\" (UID: \"456cd2fa-e3a0-40f3-b865-15d505e82bbe\") " pod="openshift-dns/node-resolver-g7nkb" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052792 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-hostroot\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052848 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-hostroot\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052901 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-var-lib-cni-bin\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.054894 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052963 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/499fd8cb-4e54-4820-a90d-c55159a3910a-os-release\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.052974 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/159d239e-3d57-4384-95cd-e37f9bdcc401-ovnkube-config\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053002 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-lib-modules\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053036 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-dbus\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053065 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-socket-dir\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053101 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cz4j\" (UniqueName: \"kubernetes.io/projected/456cd2fa-e3a0-40f3-b865-15d505e82bbe-kube-api-access-7cz4j\") pod \"node-resolver-g7nkb\" (UID: \"456cd2fa-e3a0-40f3-b865-15d505e82bbe\") " pod="openshift-dns/node-resolver-g7nkb" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053130 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-etc-openvswitch\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053181 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-etc-openvswitch\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053212 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-run-openvswitch\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053224 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-dbus\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053236 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3183f94d-897c-4803-af1c-b40745a1268d-socket-dir\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053250 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-run-openvswitch\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053257 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/499fd8cb-4e54-4820-a90d-c55159a3910a-system-cni-dir\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053297 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/499fd8cb-4e54-4820-a90d-c55159a3910a-system-cni-dir\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053304 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/499fd8cb-4e54-4820-a90d-c55159a3910a-cnibin\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053328 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-run-netns\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053351 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-run-multus-certs\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.055751 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053370 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/499fd8cb-4e54-4820-a90d-c55159a3910a-cnibin\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053408 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-run-netns\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053412 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-run-multus-certs\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053374 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-run\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053424 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-run\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-node-log\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053476 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-run-ovn-kubernetes\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053485 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-node-log\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/159d239e-3d57-4384-95cd-e37f9bdcc401-ovnkube-script-lib\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053534 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-system-cni-dir\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053558 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-var-lib-kubelet\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053582 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb493400-515b-4ed8-934f-038f1aae45ed-agent-certs\") pod \"konnectivity-agent-lbc2g\" (UID: \"fb493400-515b-4ed8-934f-038f1aae45ed\") " pod="kube-system/konnectivity-agent-lbc2g" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053603 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-run-ovn-kubernetes\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053605 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-multus-conf-dir\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053636 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-multus-conf-dir\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053679 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-systemd\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053704 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-host\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.056592 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053710 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-host-var-lib-kubelet\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.053723 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053754 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-systemd\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.053787 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret podName:12a0c5b8-e1de-44de-a231-5a56fa8a1d23 nodeName:}" failed. No retries permitted until 2026-07-10 17:59:59.55376945 +0000 UTC m=+3.100232570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret") pod "global-pull-secret-syncer-g8xxt" (UID: "12a0c5b8-e1de-44de-a231-5a56fa8a1d23") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053796 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-host\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053680 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-system-cni-dir\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053851 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frw6\" (UniqueName: \"kubernetes.io/projected/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-kube-api-access-4frw6\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-run-systemd\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053893 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-var-lib-openvswitch\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053927 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/499fd8cb-4e54-4820-a90d-c55159a3910a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053949 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8dkz\" (UniqueName: \"kubernetes.io/projected/499fd8cb-4e54-4820-a90d-c55159a3910a-kube-api-access-d8dkz\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-etc-kubernetes\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.053999 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.054004 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-var-lib-openvswitch\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.054051 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-run-systemd\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.054071 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/159d239e-3d57-4384-95cd-e37f9bdcc401-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.054109 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/159d239e-3d57-4384-95cd-e37f9bdcc401-ovnkube-script-lib\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.057380 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.054234 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d2dd6c1-df2f-4087-b03f-c18d9418c332-etc-kubernetes\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.058169 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.054429 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/159d239e-3d57-4384-95cd-e37f9bdcc401-ovn-node-metrics-cert\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.058169 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.054478 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/499fd8cb-4e54-4820-a90d-c55159a3910a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.058169 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.055886 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-etc-tuned\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.058169 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.056301 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-tmp\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.058169 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.056608 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb493400-515b-4ed8-934f-038f1aae45ed-agent-certs\") pod \"konnectivity-agent-lbc2g\" (UID: \"fb493400-515b-4ed8-934f-038f1aae45ed\") " pod="kube-system/konnectivity-agent-lbc2g" Jul 10 17:59:59.058444 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.058425 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:59:59.058499 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.058447 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:59:59.058499 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.058460 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vbhqk for pod openshift-network-diagnostics/network-check-target-z7dtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:59:59.058598 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.058514 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk podName:9766e246-9e80-449e-b500-e290fa4ac5c1 nodeName:}" failed. No retries permitted until 2026-07-10 17:59:59.558498111 +0000 UTC m=+3.104961248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vbhqk" (UniqueName: "kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk") pod "network-check-target-z7dtl" (UID: "9766e246-9e80-449e-b500-e290fa4ac5c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:59:59.059849 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.059830 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5bcb\" (UniqueName: \"kubernetes.io/projected/2d2dd6c1-df2f-4087-b03f-c18d9418c332-kube-api-access-p5bcb\") pod \"multus-qp8r9\" (UID: \"2d2dd6c1-df2f-4087-b03f-c18d9418c332\") " pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.060785 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.060765 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27v82\" (UniqueName: \"kubernetes.io/projected/e7d64404-7c29-458a-bc00-2a745b9cbc85-kube-api-access-27v82\") pod \"node-ca-f4xrb\" (UID: \"e7d64404-7c29-458a-bc00-2a745b9cbc85\") " pod="openshift-image-registry/node-ca-f4xrb" Jul 10 17:59:59.061279 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.061256 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdhzs\" (UniqueName: \"kubernetes.io/projected/c2890ff6-6c54-44b9-94b4-590301075523-kube-api-access-tdhzs\") pod \"iptables-alerter-4pm4x\" (UID: \"c2890ff6-6c54-44b9-94b4-590301075523\") " pod="openshift-network-operator/iptables-alerter-4pm4x" Jul 10 17:59:59.061436 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.061419 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqp92\" (UniqueName: \"kubernetes.io/projected/3183f94d-897c-4803-af1c-b40745a1268d-kube-api-access-kqp92\") pod \"aws-ebs-csi-driver-node-p4qb4\" (UID: \"3183f94d-897c-4803-af1c-b40745a1268d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.065653 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.065632 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cz4j\" (UniqueName: \"kubernetes.io/projected/456cd2fa-e3a0-40f3-b865-15d505e82bbe-kube-api-access-7cz4j\") pod \"node-resolver-g7nkb\" (UID: \"456cd2fa-e3a0-40f3-b865-15d505e82bbe\") " pod="openshift-dns/node-resolver-g7nkb" Jul 10 17:59:59.065952 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.065933 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ts8\" (UniqueName: \"kubernetes.io/projected/159d239e-3d57-4384-95cd-e37f9bdcc401-kube-api-access-56ts8\") pod \"ovnkube-node-6z78q\" (UID: \"159d239e-3d57-4384-95cd-e37f9bdcc401\") " pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.066044 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.065934 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frw6\" (UniqueName: \"kubernetes.io/projected/5e75bda2-8d34-49c0-bdd7-64cdd48f7801-kube-api-access-4frw6\") pod \"tuned-5p5nk\" (UID: \"5e75bda2-8d34-49c0-bdd7-64cdd48f7801\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.066044 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.066033 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8dkz\" (UniqueName: \"kubernetes.io/projected/499fd8cb-4e54-4820-a90d-c55159a3910a-kube-api-access-d8dkz\") pod \"multus-additional-cni-plugins-dmqqt\" (UID: \"499fd8cb-4e54-4820-a90d-c55159a3910a\") " pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.157348 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.157261 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4pm4x" Jul 10 17:59:59.165405 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.165381 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 17:59:59.173364 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.173343 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lbc2g" Jul 10 17:59:59.180108 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.180085 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" Jul 10 17:59:59.186743 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.186722 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" Jul 10 17:59:59.192768 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.192747 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g7nkb" Jul 10 17:59:59.200493 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.200475 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f4xrb" Jul 10 17:59:59.208254 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.208221 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dmqqt" Jul 10 17:59:59.213889 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.213875 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qp8r9" Jul 10 17:59:59.456687 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.456600 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs\") pod \"network-metrics-daemon-z5g5q\" (UID: \"c1121149-7010-4fb2-a378-dc79d124db58\") " pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 17:59:59.456865 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.456761 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:59:59.456865 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.456833 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs podName:c1121149-7010-4fb2-a378-dc79d124db58 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:00.456818988 +0000 UTC m=+4.003282121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs") pod "network-metrics-daemon-z5g5q" (UID: "c1121149-7010-4fb2-a378-dc79d124db58") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:59:59.557587 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.557549 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 17:59:59.557739 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.557667 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:59:59.557786 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.557743 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret podName:12a0c5b8-e1de-44de-a231-5a56fa8a1d23 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:00.557723528 +0000 UTC m=+4.104186659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret") pod "global-pull-secret-syncer-g8xxt" (UID: "12a0c5b8-e1de-44de-a231-5a56fa8a1d23") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:59:59.658609 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.658567 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhqk\" (UniqueName: \"kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk\") pod \"network-check-target-z7dtl\" (UID: \"9766e246-9e80-449e-b500-e290fa4ac5c1\") " pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 17:59:59.658786 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.658766 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:59:59.658846 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.658790 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:59:59.658846 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.658801 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vbhqk for pod openshift-network-diagnostics/network-check-target-z7dtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:59:59.658931 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.658885 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk podName:9766e246-9e80-449e-b500-e290fa4ac5c1 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:00.658865846 +0000 UTC m=+4.205328970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbhqk" (UniqueName: "kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk") pod "network-check-target-z7dtl" (UID: "9766e246-9e80-449e-b500-e290fa4ac5c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:59:59.874027 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:59.873924 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3183f94d_897c_4803_af1c_b40745a1268d.slice/crio-0988d569f0eb2b2017776a9482f2283472a7f8c9895fe8a85cbc25353a332540 WatchSource:0}: Error finding container 0988d569f0eb2b2017776a9482f2283472a7f8c9895fe8a85cbc25353a332540: Status 404 returned error can't find the container with id 0988d569f0eb2b2017776a9482f2283472a7f8c9895fe8a85cbc25353a332540 Jul 10 17:59:59.875406 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:59.875384 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e75bda2_8d34_49c0_bdd7_64cdd48f7801.slice/crio-a4b7c80e0a60edfc67e3bd420b80e5d19e76d72ac5c076096902a7c02b40c2c8 WatchSource:0}: Error finding container a4b7c80e0a60edfc67e3bd420b80e5d19e76d72ac5c076096902a7c02b40c2c8: Status 404 returned error can't find the container with id a4b7c80e0a60edfc67e3bd420b80e5d19e76d72ac5c076096902a7c02b40c2c8 Jul 10 17:59:59.876303 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:59.876184 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d64404_7c29_458a_bc00_2a745b9cbc85.slice/crio-ec09fb5b8f32d6179754175c0dd28751c9024a88b673f3ef36658729bbeeb488 WatchSource:0}: Error finding container ec09fb5b8f32d6179754175c0dd28751c9024a88b673f3ef36658729bbeeb488: Status 404 returned error can't find the container with id ec09fb5b8f32d6179754175c0dd28751c9024a88b673f3ef36658729bbeeb488 Jul 10 17:59:59.879984 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:59.879960 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2890ff6_6c54_44b9_94b4_590301075523.slice/crio-36a3b555056b1516eacf497c6d285abb1a25daa2b64e285b15ce2714f6f5aadd WatchSource:0}: Error finding container 36a3b555056b1516eacf497c6d285abb1a25daa2b64e285b15ce2714f6f5aadd: Status 404 returned error can't find the container with id 36a3b555056b1516eacf497c6d285abb1a25daa2b64e285b15ce2714f6f5aadd Jul 10 17:59:59.880962 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:59.880939 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb493400_515b_4ed8_934f_038f1aae45ed.slice/crio-46c769fac57cbbec1335bb64c3dd7c923786baa01cb3153a63befb925542e720 WatchSource:0}: Error finding container 46c769fac57cbbec1335bb64c3dd7c923786baa01cb3153a63befb925542e720: Status 404 returned error can't find the container with id 46c769fac57cbbec1335bb64c3dd7c923786baa01cb3153a63befb925542e720 Jul 10 17:59:59.882931 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:59.882294 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456cd2fa_e3a0_40f3_b865_15d505e82bbe.slice/crio-f5a5bdba9baef8f29766cf262232ff915c2325a2db01d16ee625cceadd0c49a5 WatchSource:0}: Error finding container f5a5bdba9baef8f29766cf262232ff915c2325a2db01d16ee625cceadd0c49a5: Status 404 returned error can't find the container with id f5a5bdba9baef8f29766cf262232ff915c2325a2db01d16ee625cceadd0c49a5 Jul 10 17:59:59.883258 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:59.883168 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159d239e_3d57_4384_95cd_e37f9bdcc401.slice/crio-463f90b49a8551eb94d3aa2384f985d1b6de7ac8a0eccbc098101165d0dc56ce WatchSource:0}: Error finding container 463f90b49a8551eb94d3aa2384f985d1b6de7ac8a0eccbc098101165d0dc56ce: Status 404 returned error can't find the container with id 463f90b49a8551eb94d3aa2384f985d1b6de7ac8a0eccbc098101165d0dc56ce Jul 10 17:59:59.884120 ip-10-0-136-200 kubenswrapper[2569]: W0710 17:59:59.884085 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2dd6c1_df2f_4087_b03f_c18d9418c332.slice/crio-cfff7fd28294ba6fd385058f068091b3dee0337f2da20b37621e1ccb7763cec0 WatchSource:0}: Error finding container cfff7fd28294ba6fd385058f068091b3dee0337f2da20b37621e1ccb7763cec0: Status 404 returned error can't find the container with id cfff7fd28294ba6fd385058f068091b3dee0337f2da20b37621e1ccb7763cec0 Jul 10 17:59:59.906348 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.906304 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 17:54:58 +0000 UTC" deadline="2028-04-18 20:24:11.43339165 +0000 UTC" Jul 10 17:59:59.906348 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.906348 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15554h24m11.527047785s" Jul 10 17:59:59.978132 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.977960 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 17:59:59.978270 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.978016 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 17:59:59.978270 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.978214 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 17:59:59.978392 ip-10-0-136-200 kubenswrapper[2569]: E0710 17:59:59.978311 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 17:59:59.987789 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.987765 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmqqt" event={"ID":"499fd8cb-4e54-4820-a90d-c55159a3910a","Type":"ContainerStarted","Data":"79d0bd11d573c6fe2dbc481edb1201742af6e137948a03ce2f3b2c53ede53c3c"} Jul 10 17:59:59.990579 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.990549 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lbc2g" event={"ID":"fb493400-515b-4ed8-934f-038f1aae45ed","Type":"ContainerStarted","Data":"46c769fac57cbbec1335bb64c3dd7c923786baa01cb3153a63befb925542e720"} Jul 10 17:59:59.992032 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.992009 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4pm4x" event={"ID":"c2890ff6-6c54-44b9-94b4-590301075523","Type":"ContainerStarted","Data":"36a3b555056b1516eacf497c6d285abb1a25daa2b64e285b15ce2714f6f5aadd"} Jul 10 17:59:59.994552 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.994531 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" event={"ID":"5e75bda2-8d34-49c0-bdd7-64cdd48f7801","Type":"ContainerStarted","Data":"a4b7c80e0a60edfc67e3bd420b80e5d19e76d72ac5c076096902a7c02b40c2c8"} Jul 10 17:59:59.997119 ip-10-0-136-200 kubenswrapper[2569]: I0710 17:59:59.997098 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-200.ec2.internal" event={"ID":"4f403d9863d495b5a87ed67af63e35db","Type":"ContainerStarted","Data":"f405c6ae4c1f08aea351cf31528db2f49af63244bbea31d560f6df0f1b0b159d"} Jul 10 18:00:00.000964 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:00.000942 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qp8r9" event={"ID":"2d2dd6c1-df2f-4087-b03f-c18d9418c332","Type":"ContainerStarted","Data":"cfff7fd28294ba6fd385058f068091b3dee0337f2da20b37621e1ccb7763cec0"} Jul 10 18:00:00.002521 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:00.002494 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" event={"ID":"159d239e-3d57-4384-95cd-e37f9bdcc401","Type":"ContainerStarted","Data":"463f90b49a8551eb94d3aa2384f985d1b6de7ac8a0eccbc098101165d0dc56ce"} Jul 10 18:00:00.003928 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:00.003902 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g7nkb" event={"ID":"456cd2fa-e3a0-40f3-b865-15d505e82bbe","Type":"ContainerStarted","Data":"f5a5bdba9baef8f29766cf262232ff915c2325a2db01d16ee625cceadd0c49a5"} Jul 10 18:00:00.004847 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:00.004828 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f4xrb" event={"ID":"e7d64404-7c29-458a-bc00-2a745b9cbc85","Type":"ContainerStarted","Data":"ec09fb5b8f32d6179754175c0dd28751c9024a88b673f3ef36658729bbeeb488"} Jul 10 18:00:00.005768 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:00.005743 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" event={"ID":"3183f94d-897c-4803-af1c-b40745a1268d","Type":"ContainerStarted","Data":"0988d569f0eb2b2017776a9482f2283472a7f8c9895fe8a85cbc25353a332540"} Jul 10 18:00:00.465545 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:00.465504 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs\") pod \"network-metrics-daemon-z5g5q\" (UID: \"c1121149-7010-4fb2-a378-dc79d124db58\") " pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:00.465713 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:00.465677 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:00:00.465773 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:00.465740 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs podName:c1121149-7010-4fb2-a378-dc79d124db58 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:02.46572118 +0000 UTC m=+6.012184304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs") pod "network-metrics-daemon-z5g5q" (UID: "c1121149-7010-4fb2-a378-dc79d124db58") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:00:00.567084 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:00.566527 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:00.567084 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:00.566665 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:00:00.567084 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:00.566736 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret podName:12a0c5b8-e1de-44de-a231-5a56fa8a1d23 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:02.566718342 +0000 UTC m=+6.113181469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret") pod "global-pull-secret-syncer-g8xxt" (UID: "12a0c5b8-e1de-44de-a231-5a56fa8a1d23") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:00:00.632636 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:00.632367 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:00:00.668265 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:00.667656 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhqk\" (UniqueName: \"kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk\") pod \"network-check-target-z7dtl\" (UID: \"9766e246-9e80-449e-b500-e290fa4ac5c1\") " pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:00.668265 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:00.667842 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:00:00.668265 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:00.667861 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:00:00.668265 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:00.667875 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vbhqk for pod openshift-network-diagnostics/network-check-target-z7dtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:00:00.668265 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:00.667939 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk podName:9766e246-9e80-449e-b500-e290fa4ac5c1 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:02.667920005 +0000 UTC m=+6.214383130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbhqk" (UniqueName: "kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk") pod "network-check-target-z7dtl" (UID: "9766e246-9e80-449e-b500-e290fa4ac5c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:00:00.906942 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:00.906680 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 17:54:58 +0000 UTC" deadline="2028-04-23 18:53:00.77514835 +0000 UTC" Jul 10 18:00:00.906942 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:00.906721 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15672h52m59.868431959s" Jul 10 18:00:00.983384 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:00.979403 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:00.983384 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:00.979549 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:01.015552 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:01.014391 2569 generic.go:356] "Generic (PLEG): container finished" podID="4643467acf069073b49fa3fc327d952b" containerID="3dccfab68f33317b9d6f30bff9ea9169600e1d5b3248f2accef11cbf8cbdb1e5" exitCode=0 Jul 10 18:00:01.015552 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:01.015054 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal" event={"ID":"4643467acf069073b49fa3fc327d952b","Type":"ContainerDied","Data":"3dccfab68f33317b9d6f30bff9ea9169600e1d5b3248f2accef11cbf8cbdb1e5"} Jul 10 18:00:01.030547 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:01.030494 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-200.ec2.internal" podStartSLOduration=4.030475188 podStartE2EDuration="4.030475188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:00.009556232 +0000 UTC m=+3.556019373" watchObservedRunningTime="2026-07-10 18:00:01.030475188 +0000 UTC m=+4.576938331" Jul 10 18:00:01.978473 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:01.978432 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:01.978976 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:01.978573 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:01.979059 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:01.979017 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:01.979270 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:01.979113 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:02.043150 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:02.042638 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal" event={"ID":"4643467acf069073b49fa3fc327d952b","Type":"ContainerStarted","Data":"f3e0d27ff72c1fc64bc1bec742e52bd9a04d9372ca3eb11bdef49e3c76538451"} Jul 10 18:00:02.486158 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:02.486095 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs\") pod \"network-metrics-daemon-z5g5q\" (UID: \"c1121149-7010-4fb2-a378-dc79d124db58\") " pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:02.486354 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:02.486301 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:00:02.486416 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:02.486363 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs podName:c1121149-7010-4fb2-a378-dc79d124db58 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:06.486344934 +0000 UTC m=+10.032808059 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs") pod "network-metrics-daemon-z5g5q" (UID: "c1121149-7010-4fb2-a378-dc79d124db58") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:00:02.587934 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:02.587229 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:02.587934 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:02.587483 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:00:02.587934 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:02.587549 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret podName:12a0c5b8-e1de-44de-a231-5a56fa8a1d23 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:06.587530614 +0000 UTC m=+10.133993737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret") pod "global-pull-secret-syncer-g8xxt" (UID: "12a0c5b8-e1de-44de-a231-5a56fa8a1d23") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:00:02.687935 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:02.687899 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhqk\" (UniqueName: \"kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk\") pod \"network-check-target-z7dtl\" (UID: \"9766e246-9e80-449e-b500-e290fa4ac5c1\") " pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:02.688118 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:02.688102 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:00:02.688197 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:02.688127 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:00:02.688197 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:02.688142 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vbhqk for pod openshift-network-diagnostics/network-check-target-z7dtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:00:02.688295 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:02.688198 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk podName:9766e246-9e80-449e-b500-e290fa4ac5c1 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:06.688180386 +0000 UTC m=+10.234643511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbhqk" (UniqueName: "kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk") pod "network-check-target-z7dtl" (UID: "9766e246-9e80-449e-b500-e290fa4ac5c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:00:02.978717 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:02.978399 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:02.978717 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:02.978538 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:03.978215 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:03.978182 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:03.978377 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:03.978199 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:03.978377 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:03.978319 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:03.978783 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:03.978754 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:04.978837 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:04.978750 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:04.979261 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:04.978916 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:05.978189 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:05.978148 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:05.978336 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:05.978156 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:05.978336 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:05.978271 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:05.978462 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:05.978389 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:06.516984 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:06.516947 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs\") pod \"network-metrics-daemon-z5g5q\" (UID: \"c1121149-7010-4fb2-a378-dc79d124db58\") " pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:06.517441 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:06.517149 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:00:06.517441 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:06.517208 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs podName:c1121149-7010-4fb2-a378-dc79d124db58 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:14.517188024 +0000 UTC m=+18.063651162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs") pod "network-metrics-daemon-z5g5q" (UID: "c1121149-7010-4fb2-a378-dc79d124db58") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:00:06.617315 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:06.617269 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:06.617489 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:06.617452 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:00:06.617548 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:06.617513 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret podName:12a0c5b8-e1de-44de-a231-5a56fa8a1d23 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:14.617492942 +0000 UTC m=+18.163956082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret") pod "global-pull-secret-syncer-g8xxt" (UID: "12a0c5b8-e1de-44de-a231-5a56fa8a1d23") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:00:06.718342 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:06.718255 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhqk\" (UniqueName: \"kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk\") pod \"network-check-target-z7dtl\" (UID: \"9766e246-9e80-449e-b500-e290fa4ac5c1\") " pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:06.718506 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:06.718444 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:00:06.718506 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:06.718470 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:00:06.718506 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:06.718485 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vbhqk for pod openshift-network-diagnostics/network-check-target-z7dtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:00:06.718665 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:06.718542 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk podName:9766e246-9e80-449e-b500-e290fa4ac5c1 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:14.718525626 +0000 UTC m=+18.264988760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbhqk" (UniqueName: "kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk") pod "network-check-target-z7dtl" (UID: "9766e246-9e80-449e-b500-e290fa4ac5c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:00:06.979315 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:06.979212 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:06.979452 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:06.979327 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:07.978957 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:07.978535 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:07.978957 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:07.978665 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:07.978957 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:07.978950 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:07.979534 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:07.979058 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:08.978283 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:08.978238 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:08.978462 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:08.978390 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:09.978882 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:09.978846 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:09.978882 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:09.978867 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:09.979310 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:09.978957 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:09.979310 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:09.979090 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:10.978480 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:10.978447 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:10.978651 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:10.978586 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:11.978539 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:11.978499 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:11.979196 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:11.978626 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:11.979196 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:11.978501 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:11.979196 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:11.978948 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:12.978906 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:12.978864 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:12.979354 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:12.979010 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:13.978090 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:13.978057 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:13.978310 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:13.978202 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:13.978310 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:13.978243 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:13.978396 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:13.978312 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:14.580468 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:14.578778 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs\") pod \"network-metrics-daemon-z5g5q\" (UID: \"c1121149-7010-4fb2-a378-dc79d124db58\") " pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:14.580468 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:14.578962 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:00:14.580468 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:14.579063 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs podName:c1121149-7010-4fb2-a378-dc79d124db58 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:30.579037869 +0000 UTC m=+34.125501006 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs") pod "network-metrics-daemon-z5g5q" (UID: "c1121149-7010-4fb2-a378-dc79d124db58") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:00:14.679534 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:14.679495 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:14.679722 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:14.679658 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:00:14.679789 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:14.679733 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret podName:12a0c5b8-e1de-44de-a231-5a56fa8a1d23 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:30.67971637 +0000 UTC m=+34.226179513 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret") pod "global-pull-secret-syncer-g8xxt" (UID: "12a0c5b8-e1de-44de-a231-5a56fa8a1d23") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:00:14.780522 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:14.780492 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhqk\" (UniqueName: \"kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk\") pod \"network-check-target-z7dtl\" (UID: \"9766e246-9e80-449e-b500-e290fa4ac5c1\") " pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:14.780706 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:14.780682 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:00:14.780756 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:14.780708 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:00:14.780756 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:14.780722 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vbhqk for pod openshift-network-diagnostics/network-check-target-z7dtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:00:14.780843 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:14.780780 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk podName:9766e246-9e80-449e-b500-e290fa4ac5c1 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:30.780763014 +0000 UTC m=+34.327226148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbhqk" (UniqueName: "kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk") pod "network-check-target-z7dtl" (UID: "9766e246-9e80-449e-b500-e290fa4ac5c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:00:14.978721 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:14.978643 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:14.978882 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:14.978766 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:15.978518 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:15.978487 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:15.978951 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:15.978496 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:15.978951 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:15.978586 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:15.978951 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:15.978696 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:16.979724 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:16.979692 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:16.980185 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:16.979829 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:17.978772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:17.978739 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:17.978955 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:17.978783 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:17.978955 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:17.978871 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:17.979072 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:17.978970 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:18.978866 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:18.978598 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:18.979430 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:18.979008 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:19.076338 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.076185 2569 generic.go:356] "Generic (PLEG): container finished" podID="499fd8cb-4e54-4820-a90d-c55159a3910a" containerID="a6fedc5054033c2a03318403c2fea816f4d737b426d91715ff3d4f300073fd62" exitCode=0 Jul 10 18:00:19.076338 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.076274 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmqqt" event={"ID":"499fd8cb-4e54-4820-a90d-c55159a3910a","Type":"ContainerDied","Data":"a6fedc5054033c2a03318403c2fea816f4d737b426d91715ff3d4f300073fd62"} Jul 10 18:00:19.078311 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.078090 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lbc2g" event={"ID":"fb493400-515b-4ed8-934f-038f1aae45ed","Type":"ContainerStarted","Data":"6f094bd8296c84161e8abda448d21955c14e6bee8cc5d44519ec780cfb16e56d"} Jul 10 18:00:19.079631 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.079604 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" event={"ID":"5e75bda2-8d34-49c0-bdd7-64cdd48f7801","Type":"ContainerStarted","Data":"5cf285e2df2c3755a6bad2c086364efbe0e83a84f1dd8edd1ef7785b9a68f686"} Jul 10 18:00:19.081202 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.081177 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qp8r9" event={"ID":"2d2dd6c1-df2f-4087-b03f-c18d9418c332","Type":"ContainerStarted","Data":"8206c85b6b35e85383f74da902c025023111210773dfd91d519b65abf55d09eb"} Jul 10 18:00:19.083316 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.083264 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:00:19.083582 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.083565 2569 generic.go:356] "Generic (PLEG): container finished" podID="159d239e-3d57-4384-95cd-e37f9bdcc401" containerID="a2307b677e1120796164946a5eb661b8472045e0f82bbea24b1224682537f25e" exitCode=1 Jul 10 18:00:19.083647 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.083629 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" event={"ID":"159d239e-3d57-4384-95cd-e37f9bdcc401","Type":"ContainerStarted","Data":"8484f3b77e83fdc62374ba129d64fa7bdd0f4710d5dc94874b6a81a3e3c329ef"} Jul 10 18:00:19.083697 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.083686 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" event={"ID":"159d239e-3d57-4384-95cd-e37f9bdcc401","Type":"ContainerDied","Data":"a2307b677e1120796164946a5eb661b8472045e0f82bbea24b1224682537f25e"} Jul 10 18:00:19.083766 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.083753 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" event={"ID":"159d239e-3d57-4384-95cd-e37f9bdcc401","Type":"ContainerStarted","Data":"1666d7a02a041ceabf3cbada8d19ec6910beac2b6fcb7961c54a2f72e9cec557"} Jul 10 18:00:19.084900 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.084880 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g7nkb" event={"ID":"456cd2fa-e3a0-40f3-b865-15d505e82bbe","Type":"ContainerStarted","Data":"93f7e6eb9370f551b1a67f086bc70077726a214e83fc1815731d572d6b546c86"} Jul 10 18:00:19.086226 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.086202 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f4xrb" event={"ID":"e7d64404-7c29-458a-bc00-2a745b9cbc85","Type":"ContainerStarted","Data":"5c9025855171e3ffe6dddbe2a4e4f73cc8600554674227976dd14f0fb7ab4b8b"} Jul 10 18:00:19.087499 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.087481 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" event={"ID":"3183f94d-897c-4803-af1c-b40745a1268d","Type":"ContainerStarted","Data":"ae884c873ca93c97ca0fc4a3365cac74e6af4ee1da41eff24f41501a6a57d0a7"} Jul 10 18:00:19.098722 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.098683 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-200.ec2.internal" podStartSLOduration=22.09867193 podStartE2EDuration="22.09867193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:02.060539324 +0000 UTC m=+5.607002467" watchObservedRunningTime="2026-07-10 18:00:19.09867193 +0000 UTC m=+22.645135070" Jul 10 18:00:19.113963 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.113927 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5p5nk" podStartSLOduration=3.424397637 podStartE2EDuration="22.113915202s" totalImagesPullingTime="18.689517565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:19.113503899 +0000 UTC m=+22.659967041" watchObservedRunningTime="2026-07-10 18:00:19.113915202 +0000 UTC m=+22.660378343" Jul 10 18:00:19.126014 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.125982 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g7nkb" podStartSLOduration=3.445217422 podStartE2EDuration="22.125970841s" totalImagesPullingTime="18.680753419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:19.125577395 +0000 UTC m=+22.672040539" watchObservedRunningTime="2026-07-10 18:00:19.125970841 +0000 UTC m=+22.672433983" Jul 10 18:00:19.138032 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.138000 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f4xrb" podStartSLOduration=3.449753768 podStartE2EDuration="22.137988832s" totalImagesPullingTime="18.688235064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:19.137659318 +0000 UTC m=+22.684122460" watchObservedRunningTime="2026-07-10 18:00:19.137988832 +0000 UTC m=+22.684451974" Jul 10 18:00:19.152580 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.152538 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qp8r9" podStartSLOduration=3.421540633 podStartE2EDuration="22.152527036s" totalImagesPullingTime="18.730986403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:19.152472364 +0000 UTC m=+22.698935508" watchObservedRunningTime="2026-07-10 18:00:19.152527036 +0000 UTC m=+22.698990177" Jul 10 18:00:19.165046 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.165013 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lbc2g" podStartSLOduration=8.019811437 podStartE2EDuration="22.165001403s" totalImagesPullingTime="14.145189966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:19.164602456 +0000 UTC m=+22.711065597" watchObservedRunningTime="2026-07-10 18:00:19.165001403 +0000 UTC m=+22.711464545" Jul 10 18:00:19.276770 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.276746 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lbc2g_fb493400-515b-4ed8-934f-038f1aae45ed/konnectivity-agent/0.log" limit=1024 Jul 10 18:00:19.346995 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.346916 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lbc2g_fb493400-515b-4ed8-934f-038f1aae45ed/konnectivity-agent/0.log" limit=1024 Jul 10 18:00:19.978883 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.978797 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:19.978883 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:19.978824 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:19.979303 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:19.978920 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:19.979303 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:19.979049 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:20.091723 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:20.091689 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4pm4x" event={"ID":"c2890ff6-6c54-44b9-94b4-590301075523","Type":"ContainerStarted","Data":"e42f001f3540ca96adb19ad56cf9728e7ad8642e7051d84230096114defcd4ec"} Jul 10 18:00:20.094800 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:20.094780 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:00:20.095185 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:20.095127 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" event={"ID":"159d239e-3d57-4384-95cd-e37f9bdcc401","Type":"ContainerStarted","Data":"a03f312ebf8f8839bbdd1f2cc003af253d6722866b0e8c63eb875cc73460f6a5"} Jul 10 18:00:20.095299 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:20.095202 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" event={"ID":"159d239e-3d57-4384-95cd-e37f9bdcc401","Type":"ContainerStarted","Data":"6d354e36d8e3eea0b7f8e11aaa13e70ba56798f04d633ae9245c17243428bc43"} Jul 10 18:00:20.095299 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:20.095254 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" event={"ID":"159d239e-3d57-4384-95cd-e37f9bdcc401","Type":"ContainerStarted","Data":"bec801845f57e4d6a18447189725118d0fbe3473b28cbccf7a6d84ed786cd368"} Jul 10 18:00:20.106950 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:20.106901 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4pm4x" podStartSLOduration=8.960129478 podStartE2EDuration="23.106888005s" totalImagesPullingTime="14.146758527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:20.106636716 +0000 UTC m=+23.653099848" watchObservedRunningTime="2026-07-10 18:00:20.106888005 +0000 UTC m=+23.653351146" Jul 10 18:00:20.223994 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:20.223966 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lbc2g_fb493400-515b-4ed8-934f-038f1aae45ed/konnectivity-agent/0.log" limit=1024 Jul 10 18:00:20.240591 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:20.240476 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 18:00:20.325078 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:20.325055 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lbc2g_fb493400-515b-4ed8-934f-038f1aae45ed/konnectivity-agent/0.log" limit=1024 Jul 10 18:00:20.909735 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:20.909617 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T18:00:20.240494564Z","UUID":"ea75b9a5-d8e8-4d52-88ca-10c16a7a7e21","Handler":null,"Name":"","Endpoint":""} Jul 10 18:00:20.911049 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:20.911035 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 18:00:20.911140 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:20.911065 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 18:00:20.978269 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:20.978216 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:20.978419 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:20.978355 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:21.099448 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:21.099408 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" event={"ID":"3183f94d-897c-4803-af1c-b40745a1268d","Type":"ContainerStarted","Data":"d8a5059392ba50f9397dc397af787f0b7e178f56600895c60d9ced5dea7f3654"} Jul 10 18:00:21.978261 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:21.978075 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:21.978416 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:21.978074 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:21.978416 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:21.978332 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:21.978416 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:21.978394 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:22.104671 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:22.104644 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:00:22.105088 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:22.105068 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" event={"ID":"159d239e-3d57-4384-95cd-e37f9bdcc401","Type":"ContainerStarted","Data":"ac9c0e587870da911e7916ce1fc204f65ec6139f2869329f3bf34ba1c1741a28"} Jul 10 18:00:22.106960 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:22.106936 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" event={"ID":"3183f94d-897c-4803-af1c-b40745a1268d","Type":"ContainerStarted","Data":"8ddd73a0bd07bde4d17feeaacec32fa1d324511faecde2b67f8759b63599b248"} Jul 10 18:00:22.803606 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:22.803577 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lbc2g" Jul 10 18:00:22.804514 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:22.804499 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lbc2g" Jul 10 18:00:22.819538 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:22.819492 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4qb4" podStartSLOduration=4.470238966 podStartE2EDuration="25.819477402s" totalImagesPullingTime="21.349238436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:22.124237538 +0000 UTC m=+25.670700680" watchObservedRunningTime="2026-07-10 18:00:22.819477402 +0000 UTC m=+26.365940587" Jul 10 18:00:22.978569 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:22.978540 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:22.978723 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:22.978690 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:23.109265 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:23.109187 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lbc2g" Jul 10 18:00:23.109771 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:23.109559 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lbc2g" Jul 10 18:00:23.232465 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:23.232433 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lbc2g_fb493400-515b-4ed8-934f-038f1aae45ed/konnectivity-agent/0.log" limit=1024 Jul 10 18:00:23.978467 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:23.978386 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:23.978627 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:23.978509 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:23.978627 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:23.978567 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:23.978747 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:23.978658 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:24.981706 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:24.981561 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:24.982129 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:24.981784 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:25.115611 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:25.115577 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmqqt" event={"ID":"499fd8cb-4e54-4820-a90d-c55159a3910a","Type":"ContainerStarted","Data":"8bfd9dbd7f7cbe1f4f7a8a737abd3cc05f42314f2675d0ad72aca30fdf4d0741"} Jul 10 18:00:25.119086 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:25.119067 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:00:25.119514 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:25.119432 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" event={"ID":"159d239e-3d57-4384-95cd-e37f9bdcc401","Type":"ContainerStarted","Data":"5d3e389d24983af205ceda913440b1ac421091d146e34fa8482bab51c7a47eed"} Jul 10 18:00:25.119756 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:25.119736 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 18:00:25.119869 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:25.119821 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 18:00:25.119939 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:25.119897 2569 scope.go:117] "RemoveContainer" containerID="a2307b677e1120796164946a5eb661b8472045e0f82bbea24b1224682537f25e" Jul 10 18:00:25.134793 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:25.134777 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 18:00:25.134974 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:25.134960 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 18:00:25.978711 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:25.978038 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:25.978711 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:25.978174 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:25.978711 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:25.978565 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:25.978711 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:25.978667 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:26.128021 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:26.127995 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:00:26.128558 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:26.128325 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" event={"ID":"159d239e-3d57-4384-95cd-e37f9bdcc401","Type":"ContainerStarted","Data":"8297bb0d337a9e13e4aa97c8369a08978d9f3e34b42621c156f8bbdfdc3b9fba"} Jul 10 18:00:26.128558 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:26.128395 2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 18:00:26.130032 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:26.130009 2569 generic.go:356] "Generic (PLEG): container finished" podID="499fd8cb-4e54-4820-a90d-c55159a3910a" containerID="8bfd9dbd7f7cbe1f4f7a8a737abd3cc05f42314f2675d0ad72aca30fdf4d0741" exitCode=0 Jul 10 18:00:26.130117 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:26.130043 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmqqt" event={"ID":"499fd8cb-4e54-4820-a90d-c55159a3910a","Type":"ContainerDied","Data":"8bfd9dbd7f7cbe1f4f7a8a737abd3cc05f42314f2675d0ad72aca30fdf4d0741"} Jul 10 18:00:26.155532 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:26.155484 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" podStartSLOduration=10.387956153 podStartE2EDuration="29.155470692s" totalImagesPullingTime="18.767514539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:26.155153214 +0000 UTC m=+29.701616356" watchObservedRunningTime="2026-07-10 18:00:26.155470692 +0000 UTC m=+29.701933833" Jul 10 18:00:26.886298 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:26.886265 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g8xxt"] Jul 10 18:00:26.886450 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:26.886380 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:26.886500 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:26.886461 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:26.889866 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:26.889824 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7dtl"] Jul 10 18:00:26.890014 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:26.889924 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:26.890096 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:26.890024 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:26.890593 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:26.890570 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z5g5q"] Jul 10 18:00:26.890712 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:26.890660 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:26.890798 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:26.890769 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:27.132480 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:27.132301 2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 18:00:27.259313 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:27.259283 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 18:00:28.136674 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:28.136642 2569 generic.go:356] "Generic (PLEG): container finished" podID="499fd8cb-4e54-4820-a90d-c55159a3910a" containerID="c9b99d376d64e0a4b85ee783fe7748421fddc1d67b029660cd6e77b237610423" exitCode=0 Jul 10 18:00:28.137172 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:28.136739 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmqqt" event={"ID":"499fd8cb-4e54-4820-a90d-c55159a3910a","Type":"ContainerDied","Data":"c9b99d376d64e0a4b85ee783fe7748421fddc1d67b029660cd6e77b237610423"} Jul 10 18:00:28.978193 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:28.978167 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:28.978193 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:28.978183 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:28.978375 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:28.978167 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:28.978375 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:28.978269 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:28.978375 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:28.978337 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:28.978486 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:28.978413 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:30.005577 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:30.005544 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lbc2g_fb493400-515b-4ed8-934f-038f1aae45ed/konnectivity-agent/0.log" limit=1024 Jul 10 18:00:30.102790 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:30.102755 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lbc2g_fb493400-515b-4ed8-934f-038f1aae45ed/konnectivity-agent/0.log" limit=1024 Jul 10 18:00:30.144016 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:30.143992 2569 generic.go:356] "Generic (PLEG): container finished" podID="499fd8cb-4e54-4820-a90d-c55159a3910a" containerID="0ca927473b80644af508ec98b29f465bd8993b3094ed1ed63e22d299442937ea" exitCode=0 Jul 10 18:00:30.144152 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:30.144052 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmqqt" event={"ID":"499fd8cb-4e54-4820-a90d-c55159a3910a","Type":"ContainerDied","Data":"0ca927473b80644af508ec98b29f465bd8993b3094ed1ed63e22d299442937ea"} Jul 10 18:00:30.307546 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:30.307519 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lbc2g_fb493400-515b-4ed8-934f-038f1aae45ed/konnectivity-agent/0.log" limit=1024 Jul 10 18:00:30.407803 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:30.407772 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lbc2g_fb493400-515b-4ed8-934f-038f1aae45ed/konnectivity-agent/0.log" limit=1024 Jul 10 18:00:30.606613 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:30.606517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs\") pod \"network-metrics-daemon-z5g5q\" (UID: \"c1121149-7010-4fb2-a378-dc79d124db58\") " pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:30.606767 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:30.606636 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:00:30.606767 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:30.606688 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs podName:c1121149-7010-4fb2-a378-dc79d124db58 nodeName:}" failed. No retries permitted until 2026-07-10 18:01:02.606673904 +0000 UTC m=+66.153137025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs") pod "network-metrics-daemon-z5g5q" (UID: "c1121149-7010-4fb2-a378-dc79d124db58") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:00:30.707603 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:30.707562 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:30.707780 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:30.707683 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:00:30.707780 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:30.707734 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret podName:12a0c5b8-e1de-44de-a231-5a56fa8a1d23 nodeName:}" failed. No retries permitted until 2026-07-10 18:01:02.70771884 +0000 UTC m=+66.254181978 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret") pod "global-pull-secret-syncer-g8xxt" (UID: "12a0c5b8-e1de-44de-a231-5a56fa8a1d23") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:00:30.808625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:30.808587 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhqk\" (UniqueName: \"kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk\") pod \"network-check-target-z7dtl\" (UID: \"9766e246-9e80-449e-b500-e290fa4ac5c1\") " pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:30.808800 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:30.808773 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:00:30.808944 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:30.808820 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:00:30.808944 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:30.808846 2569 projected.go:196] Error preparing data for projected volume kube-api-access-vbhqk for pod openshift-network-diagnostics/network-check-target-z7dtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:00:30.808944 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:30.808894 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk podName:9766e246-9e80-449e-b500-e290fa4ac5c1 nodeName:}" failed. No retries permitted until 2026-07-10 18:01:02.808879271 +0000 UTC m=+66.355342391 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbhqk" (UniqueName: "kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk") pod "network-check-target-z7dtl" (UID: "9766e246-9e80-449e-b500-e290fa4ac5c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:00:30.978685 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:30.978607 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:30.978685 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:30.978630 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:30.978852 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:30.978607 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:30.978852 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:30.978733 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5g5q" podUID="c1121149-7010-4fb2-a378-dc79d124db58" Jul 10 18:00:30.978852 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:30.978827 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8xxt" podUID="12a0c5b8-e1de-44de-a231-5a56fa8a1d23" Jul 10 18:00:30.978952 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:30.978874 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7dtl" podUID="9766e246-9e80-449e-b500-e290fa4ac5c1" Jul 10 18:00:32.300965 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.300933 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-200.ec2.internal" event="NodeReady" Jul 10 18:00:32.301386 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.301091 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 18:00:32.347108 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.347037 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t6b2r"] Jul 10 18:00:32.349108 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.349084 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:32.349644 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.349621 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m7jqp"] Jul 10 18:00:32.351348 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.351328 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m7jqp" Jul 10 18:00:32.352767 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.352744 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xxlp9\"" Jul 10 18:00:32.352767 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.352752 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 18:00:32.352950 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.352758 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 18:00:32.353996 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.353971 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 18:00:32.353996 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.353991 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 18:00:32.354135 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.354014 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 18:00:32.354135 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.354035 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mzh8h\"" Jul 10 18:00:32.360711 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.360603 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t6b2r"] Jul 10 18:00:32.363592 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.363571 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m7jqp"] Jul 10 18:00:32.519087 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.519047 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6626849-dd7f-41ff-b019-04d42ae9229d-config-volume\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:32.519268 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.519161 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:32.519268 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.519200 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6626849-dd7f-41ff-b019-04d42ae9229d-tmp-dir\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:32.519369 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.519265 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7ncj\" (UniqueName: \"kubernetes.io/projected/b6626849-dd7f-41ff-b019-04d42ae9229d-kube-api-access-p7ncj\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:32.519369 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.519304 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr6rx\" (UniqueName: \"kubernetes.io/projected/c68aaf65-4c9a-4016-9200-088deed583b9-kube-api-access-nr6rx\") pod \"ingress-canary-m7jqp\" (UID: \"c68aaf65-4c9a-4016-9200-088deed583b9\") " pod="openshift-ingress-canary/ingress-canary-m7jqp" Jul 10 18:00:32.519440 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.519396 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c68aaf65-4c9a-4016-9200-088deed583b9-cert\") pod \"ingress-canary-m7jqp\" (UID: \"c68aaf65-4c9a-4016-9200-088deed583b9\") " pod="openshift-ingress-canary/ingress-canary-m7jqp" Jul 10 18:00:32.619868 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.619787 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c68aaf65-4c9a-4016-9200-088deed583b9-cert\") pod \"ingress-canary-m7jqp\" (UID: \"c68aaf65-4c9a-4016-9200-088deed583b9\") " pod="openshift-ingress-canary/ingress-canary-m7jqp" Jul 10 18:00:32.619868 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.619838 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6626849-dd7f-41ff-b019-04d42ae9229d-config-volume\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:32.620082 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.619891 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:32.620082 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.619910 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6626849-dd7f-41ff-b019-04d42ae9229d-tmp-dir\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:32.620270 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.620247 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7ncj\" (UniqueName: \"kubernetes.io/projected/b6626849-dd7f-41ff-b019-04d42ae9229d-kube-api-access-p7ncj\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:32.620270 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.620262 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6626849-dd7f-41ff-b019-04d42ae9229d-tmp-dir\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:32.620408 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.620287 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr6rx\" (UniqueName: \"kubernetes.io/projected/c68aaf65-4c9a-4016-9200-088deed583b9-kube-api-access-nr6rx\") pod \"ingress-canary-m7jqp\" (UID: \"c68aaf65-4c9a-4016-9200-088deed583b9\") " pod="openshift-ingress-canary/ingress-canary-m7jqp" Jul 10 18:00:32.620408 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:32.620252 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:00:32.620408 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:32.620352 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls podName:b6626849-dd7f-41ff-b019-04d42ae9229d nodeName:}" failed. No retries permitted until 2026-07-10 18:00:33.120334381 +0000 UTC m=+36.666797515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls") pod "dns-default-t6b2r" (UID: "b6626849-dd7f-41ff-b019-04d42ae9229d") : secret "dns-default-metrics-tls" not found Jul 10 18:00:32.620615 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.620578 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6626849-dd7f-41ff-b019-04d42ae9229d-config-volume\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:32.624497 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.624475 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c68aaf65-4c9a-4016-9200-088deed583b9-cert\") pod \"ingress-canary-m7jqp\" (UID: \"c68aaf65-4c9a-4016-9200-088deed583b9\") " pod="openshift-ingress-canary/ingress-canary-m7jqp" Jul 10 18:00:32.630262 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.630089 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7ncj\" (UniqueName: \"kubernetes.io/projected/b6626849-dd7f-41ff-b019-04d42ae9229d-kube-api-access-p7ncj\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:32.630393 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.630373 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr6rx\" (UniqueName: \"kubernetes.io/projected/c68aaf65-4c9a-4016-9200-088deed583b9-kube-api-access-nr6rx\") pod \"ingress-canary-m7jqp\" (UID: \"c68aaf65-4c9a-4016-9200-088deed583b9\") " pod="openshift-ingress-canary/ingress-canary-m7jqp" Jul 10 18:00:32.669201 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.669181 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m7jqp" Jul 10 18:00:32.806918 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.806888 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m7jqp"] Jul 10 18:00:32.810320 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:00:32.810289 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68aaf65_4c9a_4016_9200_088deed583b9.slice/crio-95eb622f5258e68b08d8279b407237de6641a348c989429ed1bb6411644a6765 WatchSource:0}: Error finding container 95eb622f5258e68b08d8279b407237de6641a348c989429ed1bb6411644a6765: Status 404 returned error can't find the container with id 95eb622f5258e68b08d8279b407237de6641a348c989429ed1bb6411644a6765 Jul 10 18:00:32.978253 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.978143 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:00:32.978253 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.978169 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:00:32.978253 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.978143 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:00:32.981650 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.981377 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 18:00:32.981650 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.981416 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4tg5\"" Jul 10 18:00:32.981650 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.981451 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 18:00:32.981650 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.981527 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 18:00:32.981650 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.981415 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-25vgp\"" Jul 10 18:00:32.981650 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:32.981376 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 18:00:33.125896 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.125857 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:33.126059 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:33.126014 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:00:33.126123 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:33.126083 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls podName:b6626849-dd7f-41ff-b019-04d42ae9229d nodeName:}" failed. No retries permitted until 2026-07-10 18:00:34.126062782 +0000 UTC m=+37.672525910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls") pod "dns-default-t6b2r" (UID: "b6626849-dd7f-41ff-b019-04d42ae9229d") : secret "dns-default-metrics-tls" not found Jul 10 18:00:33.151241 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.151212 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m7jqp" event={"ID":"c68aaf65-4c9a-4016-9200-088deed583b9","Type":"ContainerStarted","Data":"95eb622f5258e68b08d8279b407237de6641a348c989429ed1bb6411644a6765"} Jul 10 18:00:33.729689 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.729655 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gzbnj"] Jul 10 18:00:33.732514 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.732489 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gzbnj" Jul 10 18:00:33.735410 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.735375 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 10 18:00:33.735530 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.735472 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:00:33.736478 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.736458 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-29jlw\"" Jul 10 18:00:33.742494 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.742441 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gzbnj"] Jul 10 18:00:33.831789 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.831736 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgrt8\" (UniqueName: \"kubernetes.io/projected/c81d7714-2c32-4459-9221-41271562f618-kube-api-access-dgrt8\") pod \"volume-data-source-validator-78495597f6-gzbnj\" (UID: \"c81d7714-2c32-4459-9221-41271562f618\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gzbnj" Jul 10 18:00:33.833761 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.833735 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-x72w7"] Jul 10 18:00:33.836005 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.835972 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:33.838468 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.838299 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 10 18:00:33.839243 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.839042 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 10 18:00:33.839419 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.839397 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-z6w98\"" Jul 10 18:00:33.839505 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.839342 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 10 18:00:33.839562 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.839350 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:00:33.839647 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.839373 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv"] Jul 10 18:00:33.842208 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.842187 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-lt7vz"] Jul 10 18:00:33.842370 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.842353 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:00:33.844061 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.844037 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5b875d77bd-5c2fl"] Jul 10 18:00:33.844538 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.844521 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:33.845522 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.844758 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:00:33.845522 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.845322 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 10 18:00:33.845522 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.845425 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9kxw5\"" Jul 10 18:00:33.845522 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.845511 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 10 18:00:33.846828 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.846791 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:33.847234 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.847115 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 10 18:00:33.847578 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.847427 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 10 18:00:33.847578 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.847470 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 18:00:33.847755 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.847670 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 18:00:33.847755 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.847701 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5csm8\"" Jul 10 18:00:33.848308 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.848071 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 10 18:00:33.849698 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.849498 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 10 18:00:33.850064 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.850046 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-x72w7"] Jul 10 18:00:33.850315 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.850288 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 18:00:33.850552 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.850533 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 10 18:00:33.850836 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.850783 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-h58f6\"" Jul 10 18:00:33.850998 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.850980 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 10 18:00:33.851239 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.851221 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 18:00:33.851848 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.851438 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 10 18:00:33.855038 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.855009 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv"] Jul 10 18:00:33.856221 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.856203 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 10 18:00:33.856511 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.856496 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-lt7vz"] Jul 10 18:00:33.859526 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.859509 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b875d77bd-5c2fl"] Jul 10 18:00:33.861432 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.861417 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-2ptnr"] Jul 10 18:00:33.871676 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.871653 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2ptnr" Jul 10 18:00:33.873797 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.873779 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-2ptnr"] Jul 10 18:00:33.874468 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.874452 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fl4w4\"" Jul 10 18:00:33.932222 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.932186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgrt8\" (UniqueName: \"kubernetes.io/projected/c81d7714-2c32-4459-9221-41271562f618-kube-api-access-dgrt8\") pod \"volume-data-source-validator-78495597f6-gzbnj\" (UID: \"c81d7714-2c32-4459-9221-41271562f618\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gzbnj" Jul 10 18:00:33.940866 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.940837 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bbbfdc745-jjjn6"] Jul 10 18:00:33.943098 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.943066 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:33.950984 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.950962 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 18:00:33.951288 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.951265 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-skqd4\"" Jul 10 18:00:33.952355 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.952217 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 18:00:33.952355 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.952225 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 18:00:33.957363 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.957341 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgrt8\" (UniqueName: \"kubernetes.io/projected/c81d7714-2c32-4459-9221-41271562f618-kube-api-access-dgrt8\") pod \"volume-data-source-validator-78495597f6-gzbnj\" (UID: \"c81d7714-2c32-4459-9221-41271562f618\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gzbnj" Jul 10 18:00:33.958172 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.958092 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 18:00:33.967009 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:33.966984 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bbbfdc745-jjjn6"] Jul 10 18:00:34.033415 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033330 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsxvf\" (UniqueName: \"kubernetes.io/projected/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-kube-api-access-hsxvf\") pod \"cluster-samples-operator-65db858894-wtdrv\" (UID: \"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:00:34.033415 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033407 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3404e0d1-d53d-4ccf-8b88-36893095d733-tmp\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.033617 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033460 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3404e0d1-d53d-4ccf-8b88-36893095d733-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.033617 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033554 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5lj\" (UniqueName: \"kubernetes.io/projected/3404e0d1-d53d-4ccf-8b88-36893095d733-kube-api-access-ft5lj\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.033719 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033642 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4f78d82-214c-4612-adc5-32497c685a36-config\") pod \"console-operator-7866497494-x72w7\" (UID: \"a4f78d82-214c-4612-adc5-32497c685a36\") " pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:34.033719 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4f78d82-214c-4612-adc5-32497c685a36-trusted-ca\") pod \"console-operator-7866497494-x72w7\" (UID: \"a4f78d82-214c-4612-adc5-32497c685a36\") " pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:34.033719 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033712 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdrv\" (UID: \"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:00:34.033890 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033740 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3404e0d1-d53d-4ccf-8b88-36893095d733-serving-cert\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.033890 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033783 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.033890 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033832 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-default-certificate\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.033890 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033861 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-246ld\" (UniqueName: \"kubernetes.io/projected/14d16fa6-3622-410f-acac-d4ddfe4704b2-kube-api-access-246ld\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.034068 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033900 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3404e0d1-d53d-4ccf-8b88-36893095d733-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.034068 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033958 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4f78d82-214c-4612-adc5-32497c685a36-serving-cert\") pod \"console-operator-7866497494-x72w7\" (UID: \"a4f78d82-214c-4612-adc5-32497c685a36\") " pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:34.034068 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.033990 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw6tl\" (UniqueName: \"kubernetes.io/projected/a4f78d82-214c-4612-adc5-32497c685a36-kube-api-access-fw6tl\") pod \"console-operator-7866497494-x72w7\" (UID: \"a4f78d82-214c-4612-adc5-32497c685a36\") " pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:34.034068 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.034014 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.034206 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.034071 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-stats-auth\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.034206 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.034098 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3404e0d1-d53d-4ccf-8b88-36893095d733-snapshots\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.034206 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.034126 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8kc4\" (UniqueName: \"kubernetes.io/projected/721e3d68-9258-4d26-989a-f780173f1356-kube-api-access-z8kc4\") pod \"network-check-source-869dcfcdcd-2ptnr\" (UID: \"721e3d68-9258-4d26-989a-f780173f1356\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2ptnr" Jul 10 18:00:34.045411 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.045379 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gzbnj" Jul 10 18:00:34.135205 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135169 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/236ea491-caa7-41b3-9bf6-ca03364826c8-ca-trust-extracted\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.135433 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-default-certificate\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.135433 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135248 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-246ld\" (UniqueName: \"kubernetes.io/projected/14d16fa6-3622-410f-acac-d4ddfe4704b2-kube-api-access-246ld\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.135433 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135289 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3404e0d1-d53d-4ccf-8b88-36893095d733-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.135433 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135317 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8kc4\" (UniqueName: \"kubernetes.io/projected/721e3d68-9258-4d26-989a-f780173f1356-kube-api-access-z8kc4\") pod \"network-check-source-869dcfcdcd-2ptnr\" (UID: \"721e3d68-9258-4d26-989a-f780173f1356\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2ptnr" Jul 10 18:00:34.135433 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135344 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3404e0d1-d53d-4ccf-8b88-36893095d733-snapshots\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.135433 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135367 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pptz7\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-kube-api-access-pptz7\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.135433 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135393 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsxvf\" (UniqueName: \"kubernetes.io/projected/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-kube-api-access-hsxvf\") pod \"cluster-samples-operator-65db858894-wtdrv\" (UID: \"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:00:34.135433 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135425 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3404e0d1-d53d-4ccf-8b88-36893095d733-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135453 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135483 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135509 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5lj\" (UniqueName: \"kubernetes.io/projected/3404e0d1-d53d-4ccf-8b88-36893095d733-kube-api-access-ft5lj\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135536 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4f78d82-214c-4612-adc5-32497c685a36-config\") pod \"console-operator-7866497494-x72w7\" (UID: \"a4f78d82-214c-4612-adc5-32497c685a36\") " pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135560 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdrv\" (UID: \"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135585 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4f78d82-214c-4612-adc5-32497c685a36-trusted-ca\") pod \"console-operator-7866497494-x72w7\" (UID: \"a4f78d82-214c-4612-adc5-32497c685a36\") " pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135615 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4f78d82-214c-4612-adc5-32497c685a36-serving-cert\") pod \"console-operator-7866497494-x72w7\" (UID: \"a4f78d82-214c-4612-adc5-32497c685a36\") " pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135644 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6tl\" (UniqueName: \"kubernetes.io/projected/a4f78d82-214c-4612-adc5-32497c685a36-kube-api-access-fw6tl\") pod \"console-operator-7866497494-x72w7\" (UID: \"a4f78d82-214c-4612-adc5-32497c685a36\") " pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135669 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135696 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-stats-auth\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135722 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/236ea491-caa7-41b3-9bf6-ca03364826c8-image-registry-private-configuration\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135751 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-bound-sa-token\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135786 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3404e0d1-d53d-4ccf-8b88-36893095d733-tmp\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135849 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135880 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-certificates\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.135961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135912 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/236ea491-caa7-41b3-9bf6-ca03364826c8-trusted-ca\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.136729 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135940 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/236ea491-caa7-41b3-9bf6-ca03364826c8-installation-pull-secrets\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.136729 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135977 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3404e0d1-d53d-4ccf-8b88-36893095d733-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.136729 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.135986 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3404e0d1-d53d-4ccf-8b88-36893095d733-serving-cert\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.136729 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.136118 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 18:00:34.136729 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.136184 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs podName:14d16fa6-3622-410f-acac-d4ddfe4704b2 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:34.636167212 +0000 UTC m=+38.182630361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs") pod "router-default-5b875d77bd-5c2fl" (UID: "14d16fa6-3622-410f-acac-d4ddfe4704b2") : secret "router-metrics-certs-default" not found Jul 10 18:00:34.136729 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.136310 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 18:00:34.136729 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.136384 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls podName:a5e7d458-2d34-4dab-90f3-cd2f0d7327a3 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:34.636366503 +0000 UTC m=+38.182829638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls") pod "cluster-samples-operator-65db858894-wtdrv" (UID: "a5e7d458-2d34-4dab-90f3-cd2f0d7327a3") : secret "samples-operator-tls" not found Jul 10 18:00:34.136729 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.136430 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3404e0d1-d53d-4ccf-8b88-36893095d733-snapshots\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.136729 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.136436 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3404e0d1-d53d-4ccf-8b88-36893095d733-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.136729 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.136735 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:00:34.137299 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.136791 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls podName:b6626849-dd7f-41ff-b019-04d42ae9229d nodeName:}" failed. No retries permitted until 2026-07-10 18:00:36.136777324 +0000 UTC m=+39.683240444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls") pod "dns-default-t6b2r" (UID: "b6626849-dd7f-41ff-b019-04d42ae9229d") : secret "dns-default-metrics-tls" not found Jul 10 18:00:34.137299 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.136732 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4f78d82-214c-4612-adc5-32497c685a36-config\") pod \"console-operator-7866497494-x72w7\" (UID: \"a4f78d82-214c-4612-adc5-32497c685a36\") " pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:34.137299 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.136947 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3404e0d1-d53d-4ccf-8b88-36893095d733-tmp\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.137299 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.137066 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle podName:14d16fa6-3622-410f-acac-d4ddfe4704b2 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:34.637051091 +0000 UTC m=+38.183514223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle") pod "router-default-5b875d77bd-5c2fl" (UID: "14d16fa6-3622-410f-acac-d4ddfe4704b2") : configmap references non-existent config key: service-ca.crt Jul 10 18:00:34.138322 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.138268 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4f78d82-214c-4612-adc5-32497c685a36-trusted-ca\") pod \"console-operator-7866497494-x72w7\" (UID: \"a4f78d82-214c-4612-adc5-32497c685a36\") " pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:34.138426 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.138414 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-default-certificate\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.139300 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.139245 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3404e0d1-d53d-4ccf-8b88-36893095d733-serving-cert\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.139391 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.139372 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-stats-auth\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.139710 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.139675 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4f78d82-214c-4612-adc5-32497c685a36-serving-cert\") pod \"console-operator-7866497494-x72w7\" (UID: \"a4f78d82-214c-4612-adc5-32497c685a36\") " pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:34.147665 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.147641 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8kc4\" (UniqueName: \"kubernetes.io/projected/721e3d68-9258-4d26-989a-f780173f1356-kube-api-access-z8kc4\") pod \"network-check-source-869dcfcdcd-2ptnr\" (UID: \"721e3d68-9258-4d26-989a-f780173f1356\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2ptnr" Jul 10 18:00:34.149045 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.149022 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-246ld\" (UniqueName: \"kubernetes.io/projected/14d16fa6-3622-410f-acac-d4ddfe4704b2-kube-api-access-246ld\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.150146 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.150120 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsxvf\" (UniqueName: \"kubernetes.io/projected/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-kube-api-access-hsxvf\") pod \"cluster-samples-operator-65db858894-wtdrv\" (UID: \"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:00:34.150490 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.150466 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6tl\" (UniqueName: \"kubernetes.io/projected/a4f78d82-214c-4612-adc5-32497c685a36-kube-api-access-fw6tl\") pod \"console-operator-7866497494-x72w7\" (UID: \"a4f78d82-214c-4612-adc5-32497c685a36\") " pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:34.151693 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.151662 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:34.152871 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.152842 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5lj\" (UniqueName: \"kubernetes.io/projected/3404e0d1-d53d-4ccf-8b88-36893095d733-kube-api-access-ft5lj\") pod \"insights-operator-68d9dd94cf-lt7vz\" (UID: \"3404e0d1-d53d-4ccf-8b88-36893095d733\") " pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.174459 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.174428 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" Jul 10 18:00:34.189564 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.189540 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2ptnr" Jul 10 18:00:34.236993 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.236949 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/236ea491-caa7-41b3-9bf6-ca03364826c8-image-registry-private-configuration\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.236993 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.236992 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-bound-sa-token\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.237215 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.237042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.237215 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.237070 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-certificates\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.237215 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.237098 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/236ea491-caa7-41b3-9bf6-ca03364826c8-trusted-ca\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.237215 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.237122 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/236ea491-caa7-41b3-9bf6-ca03364826c8-installation-pull-secrets\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.237215 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.237172 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/236ea491-caa7-41b3-9bf6-ca03364826c8-ca-trust-extracted\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.237215 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.237212 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pptz7\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-kube-api-access-pptz7\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.237633 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.237602 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:00:34.237633 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.237627 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbbfdc745-jjjn6: secret "image-registry-tls" not found Jul 10 18:00:34.237748 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.237697 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls podName:236ea491-caa7-41b3-9bf6-ca03364826c8 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:34.737675816 +0000 UTC m=+38.284138949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls") pod "image-registry-6bbbfdc745-jjjn6" (UID: "236ea491-caa7-41b3-9bf6-ca03364826c8") : secret "image-registry-tls" not found Jul 10 18:00:34.238261 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.238211 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/236ea491-caa7-41b3-9bf6-ca03364826c8-ca-trust-extracted\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.238383 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.238362 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-certificates\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.238650 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.238607 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/236ea491-caa7-41b3-9bf6-ca03364826c8-trusted-ca\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.240864 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.240801 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/236ea491-caa7-41b3-9bf6-ca03364826c8-image-registry-private-configuration\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.241361 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.241320 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/236ea491-caa7-41b3-9bf6-ca03364826c8-installation-pull-secrets\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.252563 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.252520 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-bound-sa-token\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.252748 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.252728 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pptz7\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-kube-api-access-pptz7\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.641505 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.641465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.641669 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.641523 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdrv\" (UID: \"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:00:34.641669 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.641547 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:34.641669 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.641631 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle podName:14d16fa6-3622-410f-acac-d4ddfe4704b2 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:35.641609389 +0000 UTC m=+39.188072523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle") pod "router-default-5b875d77bd-5c2fl" (UID: "14d16fa6-3622-410f-acac-d4ddfe4704b2") : configmap references non-existent config key: service-ca.crt Jul 10 18:00:34.641669 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.641667 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 18:00:34.641861 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.641667 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 18:00:34.641861 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.641732 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls podName:a5e7d458-2d34-4dab-90f3-cd2f0d7327a3 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:35.641719329 +0000 UTC m=+39.188182448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls") pod "cluster-samples-operator-65db858894-wtdrv" (UID: "a5e7d458-2d34-4dab-90f3-cd2f0d7327a3") : secret "samples-operator-tls" not found Jul 10 18:00:34.641861 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.641752 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs podName:14d16fa6-3622-410f-acac-d4ddfe4704b2 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:35.641738323 +0000 UTC m=+39.188201452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs") pod "router-default-5b875d77bd-5c2fl" (UID: "14d16fa6-3622-410f-acac-d4ddfe4704b2") : secret "router-metrics-certs-default" not found Jul 10 18:00:34.742642 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:34.742605 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:34.743070 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.742761 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:00:34.743070 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.742778 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbbfdc745-jjjn6: secret "image-registry-tls" not found Jul 10 18:00:34.743070 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:34.742847 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls podName:236ea491-caa7-41b3-9bf6-ca03364826c8 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:35.742832872 +0000 UTC m=+39.289295995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls") pod "image-registry-6bbbfdc745-jjjn6" (UID: "236ea491-caa7-41b3-9bf6-ca03364826c8") : secret "image-registry-tls" not found Jul 10 18:00:35.650348 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:35.650306 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:35.650542 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:35.650359 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdrv\" (UID: \"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:00:35.650542 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:35.650458 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:35.650542 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:35.650491 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle podName:14d16fa6-3622-410f-acac-d4ddfe4704b2 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:37.650468988 +0000 UTC m=+41.196932112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle") pod "router-default-5b875d77bd-5c2fl" (UID: "14d16fa6-3622-410f-acac-d4ddfe4704b2") : configmap references non-existent config key: service-ca.crt Jul 10 18:00:35.650542 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:35.650524 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 18:00:35.650741 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:35.650586 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 18:00:35.650741 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:35.650601 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls podName:a5e7d458-2d34-4dab-90f3-cd2f0d7327a3 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:37.650582206 +0000 UTC m=+41.197045330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls") pod "cluster-samples-operator-65db858894-wtdrv" (UID: "a5e7d458-2d34-4dab-90f3-cd2f0d7327a3") : secret "samples-operator-tls" not found Jul 10 18:00:35.650741 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:35.650632 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs podName:14d16fa6-3622-410f-acac-d4ddfe4704b2 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:37.65061746 +0000 UTC m=+41.197080584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs") pod "router-default-5b875d77bd-5c2fl" (UID: "14d16fa6-3622-410f-acac-d4ddfe4704b2") : secret "router-metrics-certs-default" not found Jul 10 18:00:35.751745 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:35.751698 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:35.752219 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:35.751913 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:00:35.752219 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:35.751937 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbbfdc745-jjjn6: secret "image-registry-tls" not found Jul 10 18:00:35.752219 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:35.752010 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls podName:236ea491-caa7-41b3-9bf6-ca03364826c8 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:37.751989728 +0000 UTC m=+41.298452870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls") pod "image-registry-6bbbfdc745-jjjn6" (UID: "236ea491-caa7-41b3-9bf6-ca03364826c8") : secret "image-registry-tls" not found Jul 10 18:00:36.154936 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:36.154906 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:36.155082 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:36.155033 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:00:36.155137 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:36.155089 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls podName:b6626849-dd7f-41ff-b019-04d42ae9229d nodeName:}" failed. No retries permitted until 2026-07-10 18:00:40.155074828 +0000 UTC m=+43.701537950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls") pod "dns-default-t6b2r" (UID: "b6626849-dd7f-41ff-b019-04d42ae9229d") : secret "dns-default-metrics-tls" not found Jul 10 18:00:36.509468 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:36.509304 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-x72w7"] Jul 10 18:00:36.512788 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:00:36.512760 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f78d82_214c_4612_adc5_32497c685a36.slice/crio-310618b0314422c8a986d6032a3728efb2ffabe3cd26cebbb69f8275eed2f208 WatchSource:0}: Error finding container 310618b0314422c8a986d6032a3728efb2ffabe3cd26cebbb69f8275eed2f208: Status 404 returned error can't find the container with id 310618b0314422c8a986d6032a3728efb2ffabe3cd26cebbb69f8275eed2f208 Jul 10 18:00:36.726776 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:36.726709 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-2ptnr"] Jul 10 18:00:36.730511 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:36.730482 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gzbnj"] Jul 10 18:00:36.731532 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:36.731496 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-lt7vz"] Jul 10 18:00:36.732729 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:00:36.732705 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721e3d68_9258_4d26_989a_f780173f1356.slice/crio-0e681f5acc099f7baa3c8e874c90d4f4f3273d28ccbfdb04c5c104dde49f030e WatchSource:0}: Error finding container 0e681f5acc099f7baa3c8e874c90d4f4f3273d28ccbfdb04c5c104dde49f030e: Status 404 returned error can't find the container with id 0e681f5acc099f7baa3c8e874c90d4f4f3273d28ccbfdb04c5c104dde49f030e Jul 10 18:00:36.733584 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:00:36.733551 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81d7714_2c32_4459_9221_41271562f618.slice/crio-5f38c7655b204dae046874f09303e86f5e4c033a31d0373c01171abda755407c WatchSource:0}: Error finding container 5f38c7655b204dae046874f09303e86f5e4c033a31d0373c01171abda755407c: Status 404 returned error can't find the container with id 5f38c7655b204dae046874f09303e86f5e4c033a31d0373c01171abda755407c Jul 10 18:00:36.734163 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:00:36.734135 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3404e0d1_d53d_4ccf_8b88_36893095d733.slice/crio-c266b1bce6c6281916036e54ad36c3c9e9eefd3c19fbcca367f0adc3cce53fb9 WatchSource:0}: Error finding container c266b1bce6c6281916036e54ad36c3c9e9eefd3c19fbcca367f0adc3cce53fb9: Status 404 returned error can't find the container with id c266b1bce6c6281916036e54ad36c3c9e9eefd3c19fbcca367f0adc3cce53fb9 Jul 10 18:00:37.162796 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:37.162761 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2ptnr" event={"ID":"721e3d68-9258-4d26-989a-f780173f1356","Type":"ContainerStarted","Data":"0e681f5acc099f7baa3c8e874c90d4f4f3273d28ccbfdb04c5c104dde49f030e"} Jul 10 18:00:37.163823 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:37.163781 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gzbnj" event={"ID":"c81d7714-2c32-4459-9221-41271562f618","Type":"ContainerStarted","Data":"5f38c7655b204dae046874f09303e86f5e4c033a31d0373c01171abda755407c"} Jul 10 18:00:37.165082 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:37.165057 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m7jqp" event={"ID":"c68aaf65-4c9a-4016-9200-088deed583b9","Type":"ContainerStarted","Data":"aa994ff52d1868e8c304c05f960e42c60504bde68cd213c12b1e2fe5615d7736"} Jul 10 18:00:37.167609 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:37.167587 2569 generic.go:356] "Generic (PLEG): container finished" podID="499fd8cb-4e54-4820-a90d-c55159a3910a" containerID="7a05c9db226b2d428715b6f2484d66ca47ac6ace6a26f444300378ea128f7c25" exitCode=0 Jul 10 18:00:37.167703 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:37.167649 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmqqt" event={"ID":"499fd8cb-4e54-4820-a90d-c55159a3910a","Type":"ContainerDied","Data":"7a05c9db226b2d428715b6f2484d66ca47ac6ace6a26f444300378ea128f7c25"} Jul 10 18:00:37.168992 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:37.168970 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-x72w7" event={"ID":"a4f78d82-214c-4612-adc5-32497c685a36","Type":"ContainerStarted","Data":"310618b0314422c8a986d6032a3728efb2ffabe3cd26cebbb69f8275eed2f208"} Jul 10 18:00:37.169951 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:37.169932 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" event={"ID":"3404e0d1-d53d-4ccf-8b88-36893095d733","Type":"ContainerStarted","Data":"c266b1bce6c6281916036e54ad36c3c9e9eefd3c19fbcca367f0adc3cce53fb9"} Jul 10 18:00:37.180430 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:37.180392 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m7jqp" podStartSLOduration=1.673456924 podStartE2EDuration="5.180378047s" totalImagesPullingTime="3.506921123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:37.179486492 +0000 UTC m=+40.725949642" watchObservedRunningTime="2026-07-10 18:00:37.180378047 +0000 UTC m=+40.726841188" Jul 10 18:00:37.668316 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:37.668279 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:37.668521 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:37.668347 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdrv\" (UID: \"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:00:37.668521 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:37.668383 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:37.668521 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:37.668494 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle podName:14d16fa6-3622-410f-acac-d4ddfe4704b2 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:41.668470116 +0000 UTC m=+45.214933256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle") pod "router-default-5b875d77bd-5c2fl" (UID: "14d16fa6-3622-410f-acac-d4ddfe4704b2") : configmap references non-existent config key: service-ca.crt Jul 10 18:00:37.668677 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:37.668555 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 18:00:37.668677 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:37.668605 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs podName:14d16fa6-3622-410f-acac-d4ddfe4704b2 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:41.668589213 +0000 UTC m=+45.215052349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs") pod "router-default-5b875d77bd-5c2fl" (UID: "14d16fa6-3622-410f-acac-d4ddfe4704b2") : secret "router-metrics-certs-default" not found Jul 10 18:00:37.668677 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:37.668674 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 18:00:37.668897 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:37.668705 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls podName:a5e7d458-2d34-4dab-90f3-cd2f0d7327a3 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:41.668694938 +0000 UTC m=+45.215158074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls") pod "cluster-samples-operator-65db858894-wtdrv" (UID: "a5e7d458-2d34-4dab-90f3-cd2f0d7327a3") : secret "samples-operator-tls" not found Jul 10 18:00:37.770138 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:37.769890 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:37.770138 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:37.770037 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:00:37.770138 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:37.770052 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbbfdc745-jjjn6: secret "image-registry-tls" not found Jul 10 18:00:37.770138 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:37.770114 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls podName:236ea491-caa7-41b3-9bf6-ca03364826c8 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:41.770097097 +0000 UTC m=+45.316560229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls") pod "image-registry-6bbbfdc745-jjjn6" (UID: "236ea491-caa7-41b3-9bf6-ca03364826c8") : secret "image-registry-tls" not found Jul 10 18:00:38.180302 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:38.180264 2569 generic.go:356] "Generic (PLEG): container finished" podID="499fd8cb-4e54-4820-a90d-c55159a3910a" containerID="702dafbd116297360446826f6b343d283d50f6ad7c70d3a67354ad8434e72ab8" exitCode=0 Jul 10 18:00:38.180776 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:38.180457 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmqqt" event={"ID":"499fd8cb-4e54-4820-a90d-c55159a3910a","Type":"ContainerDied","Data":"702dafbd116297360446826f6b343d283d50f6ad7c70d3a67354ad8434e72ab8"} Jul 10 18:00:39.184982 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:39.184928 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gzbnj" event={"ID":"c81d7714-2c32-4459-9221-41271562f618","Type":"ContainerStarted","Data":"61634df333380ac9ab0a81ee48c0dbd609290b95713fc16b41ace58bba580c35"} Jul 10 18:00:39.189311 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:39.189279 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmqqt" event={"ID":"499fd8cb-4e54-4820-a90d-c55159a3910a","Type":"ContainerStarted","Data":"043564bd6cec30d95853514a91c381fc69779a221bf44747856fc89c72616276"} Jul 10 18:00:39.199575 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:39.199523 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gzbnj" podStartSLOduration=4.45890794 podStartE2EDuration="6.199506881s" totalImagesPullingTime="1.740598941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:39.199117623 +0000 UTC m=+42.745580777" watchObservedRunningTime="2026-07-10 18:00:39.199506881 +0000 UTC m=+42.745970023" Jul 10 18:00:39.223043 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:39.222981 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dmqqt" podStartSLOduration=9.056954506 podStartE2EDuration="42.222960336s" totalImagesPullingTime="33.16600583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:39.221075257 +0000 UTC m=+42.767538399" watchObservedRunningTime="2026-07-10 18:00:39.222960336 +0000 UTC m=+42.769423483" Jul 10 18:00:40.192660 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:40.192630 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:40.193119 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:40.192828 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:00:40.193119 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:40.192938 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls podName:b6626849-dd7f-41ff-b019-04d42ae9229d nodeName:}" failed. No retries permitted until 2026-07-10 18:00:48.192916918 +0000 UTC m=+51.739380052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls") pod "dns-default-t6b2r" (UID: "b6626849-dd7f-41ff-b019-04d42ae9229d") : secret "dns-default-metrics-tls" not found Jul 10 18:00:41.705619 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:41.705592 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:41.705950 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:41.705637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdrv\" (UID: \"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:00:41.705950 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:41.705658 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:41.705950 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:41.705748 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 18:00:41.705950 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:41.705762 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 18:00:41.705950 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:41.705790 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle podName:14d16fa6-3622-410f-acac-d4ddfe4704b2 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:49.705767527 +0000 UTC m=+53.252230669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle") pod "router-default-5b875d77bd-5c2fl" (UID: "14d16fa6-3622-410f-acac-d4ddfe4704b2") : configmap references non-existent config key: service-ca.crt Jul 10 18:00:41.705950 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:41.705832 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls podName:a5e7d458-2d34-4dab-90f3-cd2f0d7327a3 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:49.705802341 +0000 UTC m=+53.252265466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls") pod "cluster-samples-operator-65db858894-wtdrv" (UID: "a5e7d458-2d34-4dab-90f3-cd2f0d7327a3") : secret "samples-operator-tls" not found Jul 10 18:00:41.705950 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:41.705851 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs podName:14d16fa6-3622-410f-acac-d4ddfe4704b2 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:49.705841197 +0000 UTC m=+53.252304319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs") pod "router-default-5b875d77bd-5c2fl" (UID: "14d16fa6-3622-410f-acac-d4ddfe4704b2") : secret "router-metrics-certs-default" not found Jul 10 18:00:41.807081 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:41.807047 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:41.807297 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:41.807174 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:00:41.807297 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:41.807185 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbbfdc745-jjjn6: secret "image-registry-tls" not found Jul 10 18:00:41.807297 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:41.807235 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls podName:236ea491-caa7-41b3-9bf6-ca03364826c8 nodeName:}" failed. No retries permitted until 2026-07-10 18:00:49.807221977 +0000 UTC m=+53.353685097 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls") pod "image-registry-6bbbfdc745-jjjn6" (UID: "236ea491-caa7-41b3-9bf6-ca03364826c8") : secret "image-registry-tls" not found Jul 10 18:00:42.203891 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:42.203852 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2ptnr" event={"ID":"721e3d68-9258-4d26-989a-f780173f1356","Type":"ContainerStarted","Data":"b683194b27ea9f660a5659028f4e592153ffea7fea6c25b14188fdd6b6afe7e6"} Jul 10 18:00:42.205620 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:42.205597 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/0.log" Jul 10 18:00:42.205737 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:42.205638 2569 generic.go:356] "Generic (PLEG): container finished" podID="a4f78d82-214c-4612-adc5-32497c685a36" containerID="b281077aeba0d9db35c79b63b5b02ab517d50aac3a829e919dd72c2233510dbb" exitCode=255 Jul 10 18:00:42.205737 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:42.205712 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-x72w7" event={"ID":"a4f78d82-214c-4612-adc5-32497c685a36","Type":"ContainerDied","Data":"b281077aeba0d9db35c79b63b5b02ab517d50aac3a829e919dd72c2233510dbb"} Jul 10 18:00:42.205882 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:42.205867 2569 scope.go:117] "RemoveContainer" containerID="b281077aeba0d9db35c79b63b5b02ab517d50aac3a829e919dd72c2233510dbb" Jul 10 18:00:42.207445 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:42.207420 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" event={"ID":"3404e0d1-d53d-4ccf-8b88-36893095d733","Type":"ContainerStarted","Data":"42ff9c0c70ef0fd1fdb81abaeae17fd1a3f2f9b35caec26c99bbe1b4c3aeba7a"} Jul 10 18:00:42.219161 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:42.219115 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-2ptnr" podStartSLOduration=4.409079279 podStartE2EDuration="9.219098861s" totalImagesPullingTime="4.810019582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:42.217762019 +0000 UTC m=+45.764225165" watchObservedRunningTime="2026-07-10 18:00:42.219098861 +0000 UTC m=+45.765562005" Jul 10 18:00:42.251968 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:42.251925 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" podStartSLOduration=4.438592695 podStartE2EDuration="9.251912905s" totalImagesPullingTime="4.81332021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:42.251095061 +0000 UTC m=+45.797558202" watchObservedRunningTime="2026-07-10 18:00:42.251912905 +0000 UTC m=+45.798376047" Jul 10 18:00:43.006605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.006516 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d86p4"] Jul 10 18:00:43.033859 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.033827 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d86p4"] Jul 10 18:00:43.034015 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.033914 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.036346 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.036313 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 18:00:43.036495 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.036459 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 18:00:43.036569 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.036557 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pcwng\"" Jul 10 18:00:43.120167 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.120130 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-data-volume\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.120350 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.120235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.120350 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.120300 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shgnv\" (UniqueName: \"kubernetes.io/projected/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-api-access-shgnv\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.120437 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.120394 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-crio-socket\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.120437 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.120421 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.212402 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.212373 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/1.log" Jul 10 18:00:43.212844 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.212828 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/0.log" Jul 10 18:00:43.212923 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.212861 2569 generic.go:356] "Generic (PLEG): container finished" podID="a4f78d82-214c-4612-adc5-32497c685a36" containerID="432d8dd062ab5bd0136ff3e7ee540d7ab2ebc0d404ad946f1181be160d0cad92" exitCode=255 Jul 10 18:00:43.212986 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.212952 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-x72w7" event={"ID":"a4f78d82-214c-4612-adc5-32497c685a36","Type":"ContainerDied","Data":"432d8dd062ab5bd0136ff3e7ee540d7ab2ebc0d404ad946f1181be160d0cad92"} Jul 10 18:00:43.213038 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.213005 2569 scope.go:117] "RemoveContainer" containerID="b281077aeba0d9db35c79b63b5b02ab517d50aac3a829e919dd72c2233510dbb" Jul 10 18:00:43.213225 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.213203 2569 scope.go:117] "RemoveContainer" containerID="432d8dd062ab5bd0136ff3e7ee540d7ab2ebc0d404ad946f1181be160d0cad92" Jul 10 18:00:43.213420 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:43.213401 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-x72w7_openshift-console-operator(a4f78d82-214c-4612-adc5-32497c685a36)\"" pod="openshift-console-operator/console-operator-7866497494-x72w7" podUID="a4f78d82-214c-4612-adc5-32497c685a36" Jul 10 18:00:43.221311 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.221291 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-crio-socket\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.221423 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.221317 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.221423 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.221363 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-data-volume\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.221423 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.221409 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.221557 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:43.221418 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:00:43.221557 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.221448 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shgnv\" (UniqueName: \"kubernetes.io/projected/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-api-access-shgnv\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.221557 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:43.221483 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm podName:ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca nodeName:}" failed. No retries permitted until 2026-07-10 18:00:43.721465743 +0000 UTC m=+47.267928863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d86p4" (UID: "ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca") : configmap "kube-rbac-proxy" not found Jul 10 18:00:43.221557 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.221541 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-crio-socket\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.221729 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:43.221551 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:00:43.221729 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:43.221629 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls podName:ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca nodeName:}" failed. No retries permitted until 2026-07-10 18:00:43.721611889 +0000 UTC m=+47.268075027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d86p4" (UID: "ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca") : secret "insights-runtime-extractor-tls" not found Jul 10 18:00:43.221795 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.221736 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-data-volume\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.232084 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.232052 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shgnv\" (UniqueName: \"kubernetes.io/projected/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-api-access-shgnv\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.725646 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.725609 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.725834 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:43.725709 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:43.725834 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:43.725770 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:00:43.725834 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:43.725790 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:00:43.725944 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:43.725859 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls podName:ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca nodeName:}" failed. No retries permitted until 2026-07-10 18:00:44.725839928 +0000 UTC m=+48.272303055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d86p4" (UID: "ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca") : secret "insights-runtime-extractor-tls" not found Jul 10 18:00:43.725944 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:43.725875 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm podName:ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca nodeName:}" failed. No retries permitted until 2026-07-10 18:00:44.725868125 +0000 UTC m=+48.272331245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d86p4" (UID: "ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca") : configmap "kube-rbac-proxy" not found Jul 10 18:00:44.152510 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:44.152474 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:44.152900 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:44.152526 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:00:44.219268 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:44.219243 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/1.log" Jul 10 18:00:44.219606 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:44.219591 2569 scope.go:117] "RemoveContainer" containerID="432d8dd062ab5bd0136ff3e7ee540d7ab2ebc0d404ad946f1181be160d0cad92" Jul 10 18:00:44.219762 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:44.219744 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-x72w7_openshift-console-operator(a4f78d82-214c-4612-adc5-32497c685a36)\"" pod="openshift-console-operator/console-operator-7866497494-x72w7" podUID="a4f78d82-214c-4612-adc5-32497c685a36" Jul 10 18:00:44.732643 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:44.732606 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:44.732827 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:44.732683 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:44.732827 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:44.732760 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:00:44.732827 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:44.732785 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:00:44.732952 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:44.732840 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm podName:ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca nodeName:}" failed. No retries permitted until 2026-07-10 18:00:46.732804824 +0000 UTC m=+50.279267947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d86p4" (UID: "ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca") : configmap "kube-rbac-proxy" not found Jul 10 18:00:44.732952 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:44.732870 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls podName:ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca nodeName:}" failed. No retries permitted until 2026-07-10 18:00:46.732851848 +0000 UTC m=+50.279314990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d86p4" (UID: "ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca") : secret "insights-runtime-extractor-tls" not found Jul 10 18:00:45.222988 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:45.222964 2569 scope.go:117] "RemoveContainer" containerID="432d8dd062ab5bd0136ff3e7ee540d7ab2ebc0d404ad946f1181be160d0cad92" Jul 10 18:00:45.223369 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:45.223127 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-x72w7_openshift-console-operator(a4f78d82-214c-4612-adc5-32497c685a36)\"" pod="openshift-console-operator/console-operator-7866497494-x72w7" podUID="a4f78d82-214c-4612-adc5-32497c685a36" Jul 10 18:00:45.800123 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:45.800098 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g7nkb_456cd2fa-e3a0-40f3-b865-15d505e82bbe/dns-node-resolver/0.log" Jul 10 18:00:46.750351 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:46.750307 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:46.750758 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:46.750394 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:46.750758 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:46.750464 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:00:46.750758 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:46.750467 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:00:46.750758 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:46.750520 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm podName:ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca nodeName:}" failed. No retries permitted until 2026-07-10 18:00:50.750505205 +0000 UTC m=+54.296968325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d86p4" (UID: "ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca") : configmap "kube-rbac-proxy" not found Jul 10 18:00:46.750758 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:46.750535 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls podName:ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca nodeName:}" failed. No retries permitted until 2026-07-10 18:00:50.750527362 +0000 UTC m=+54.296990482 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d86p4" (UID: "ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca") : secret "insights-runtime-extractor-tls" not found Jul 10 18:00:46.800708 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:46.800678 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f4xrb_e7d64404-7c29-458a-bc00-2a745b9cbc85/node-ca/0.log" Jul 10 18:00:47.600690 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:47.600660 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m7jqp_c68aaf65-4c9a-4016-9200-088deed583b9/serve-healthcheck-canary/0.log" Jul 10 18:00:48.262317 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:48.262285 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:00:48.262702 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:48.262407 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:00:48.262702 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:48.262465 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls podName:b6626849-dd7f-41ff-b019-04d42ae9229d nodeName:}" failed. No retries permitted until 2026-07-10 18:01:04.262447652 +0000 UTC m=+67.808910774 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls") pod "dns-default-t6b2r" (UID: "b6626849-dd7f-41ff-b019-04d42ae9229d") : secret "dns-default-metrics-tls" not found Jul 10 18:00:49.774412 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:49.774369 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:49.774875 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:49.774422 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdrv\" (UID: \"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:00:49.774875 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:49.774446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:00:49.774875 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:49.774540 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 18:00:49.774875 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:49.774549 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle podName:14d16fa6-3622-410f-acac-d4ddfe4704b2 nodeName:}" failed. No retries permitted until 2026-07-10 18:01:05.774527382 +0000 UTC m=+69.320990522 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle") pod "router-default-5b875d77bd-5c2fl" (UID: "14d16fa6-3622-410f-acac-d4ddfe4704b2") : configmap references non-existent config key: service-ca.crt Jul 10 18:00:49.774875 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:49.774583 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs podName:14d16fa6-3622-410f-acac-d4ddfe4704b2 nodeName:}" failed. No retries permitted until 2026-07-10 18:01:05.774571614 +0000 UTC m=+69.321034734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs") pod "router-default-5b875d77bd-5c2fl" (UID: "14d16fa6-3622-410f-acac-d4ddfe4704b2") : secret "router-metrics-certs-default" not found Jul 10 18:00:49.774875 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:49.774593 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 18:00:49.774875 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:49.774642 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls podName:a5e7d458-2d34-4dab-90f3-cd2f0d7327a3 nodeName:}" failed. No retries permitted until 2026-07-10 18:01:05.774630841 +0000 UTC m=+69.321093978 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls") pod "cluster-samples-operator-65db858894-wtdrv" (UID: "a5e7d458-2d34-4dab-90f3-cd2f0d7327a3") : secret "samples-operator-tls" not found Jul 10 18:00:49.875157 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:49.875113 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls\") pod \"image-registry-6bbbfdc745-jjjn6\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:00:49.875307 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:49.875262 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:00:49.875307 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:49.875277 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbbfdc745-jjjn6: secret "image-registry-tls" not found Jul 10 18:00:49.875398 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:49.875331 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls podName:236ea491-caa7-41b3-9bf6-ca03364826c8 nodeName:}" failed. No retries permitted until 2026-07-10 18:01:05.875313642 +0000 UTC m=+69.421776779 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls") pod "image-registry-6bbbfdc745-jjjn6" (UID: "236ea491-caa7-41b3-9bf6-ca03364826c8") : secret "image-registry-tls" not found Jul 10 18:00:50.782904 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:50.782862 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:50.783282 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:50.782969 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:50.783282 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:50.782989 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:00:50.783282 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:50.783048 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm podName:ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca nodeName:}" failed. No retries permitted until 2026-07-10 18:00:58.783032881 +0000 UTC m=+62.329496001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d86p4" (UID: "ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca") : configmap "kube-rbac-proxy" not found Jul 10 18:00:50.783282 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:50.783089 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:00:50.783282 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:50.783139 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls podName:ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca nodeName:}" failed. No retries permitted until 2026-07-10 18:00:58.783128099 +0000 UTC m=+62.329591219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d86p4" (UID: "ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca") : secret "insights-runtime-extractor-tls" not found Jul 10 18:00:56.980892 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:56.980852 2569 scope.go:117] "RemoveContainer" containerID="432d8dd062ab5bd0136ff3e7ee540d7ab2ebc0d404ad946f1181be160d0cad92" Jul 10 18:00:57.258619 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:57.258538 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:00:57.259003 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:57.258986 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/1.log" Jul 10 18:00:57.259078 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:57.259021 2569 generic.go:356] "Generic (PLEG): container finished" podID="a4f78d82-214c-4612-adc5-32497c685a36" containerID="60abe0ba19a1a0a16b4efdecec1fe04c61a29f2189d19f7709010607965d094c" exitCode=255 Jul 10 18:00:57.259117 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:57.259079 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-x72w7" event={"ID":"a4f78d82-214c-4612-adc5-32497c685a36","Type":"ContainerDied","Data":"60abe0ba19a1a0a16b4efdecec1fe04c61a29f2189d19f7709010607965d094c"} Jul 10 18:00:57.259159 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:57.259125 2569 scope.go:117] "RemoveContainer" containerID="432d8dd062ab5bd0136ff3e7ee540d7ab2ebc0d404ad946f1181be160d0cad92" Jul 10 18:00:57.259446 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:57.259426 2569 scope.go:117] "RemoveContainer" containerID="60abe0ba19a1a0a16b4efdecec1fe04c61a29f2189d19f7709010607965d094c" Jul 10 18:00:57.259614 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:57.259589 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-x72w7_openshift-console-operator(a4f78d82-214c-4612-adc5-32497c685a36)\"" pod="openshift-console-operator/console-operator-7866497494-x72w7" podUID="a4f78d82-214c-4612-adc5-32497c685a36" Jul 10 18:00:58.150384 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:58.150359 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6z78q" Jul 10 18:00:58.266543 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:58.266514 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:00:58.850996 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:58.850954 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:58.851216 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:58.851186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:00:58.851322 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:58.851308 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:00:58.851375 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:00:58.851366 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls podName:ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca nodeName:}" failed. No retries permitted until 2026-07-10 18:01:14.851353167 +0000 UTC m=+78.397816286 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d86p4" (UID: "ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca") : secret "insights-runtime-extractor-tls" not found Jul 10 18:00:58.851541 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:00:58.851523 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:01:01.675825 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.675785 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bbbfdc745-jjjn6"] Jul 10 18:01:01.676179 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:01:01.676027 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" podUID="236ea491-caa7-41b3-9bf6-ca03364826c8" Jul 10 18:01:01.750050 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.750018 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fb7478855-8rd8t"] Jul 10 18:01:01.752904 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.752887 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.765952 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.765932 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fb7478855-8rd8t"] Jul 10 18:01:01.879803 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.879764 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/94439f31-71d0-49e6-81b7-27f670e8ef16-installation-pull-secrets\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.879994 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.879971 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/94439f31-71d0-49e6-81b7-27f670e8ef16-registry-certificates\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.880050 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.880032 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/94439f31-71d0-49e6-81b7-27f670e8ef16-ca-trust-extracted\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.880087 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.880057 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/94439f31-71d0-49e6-81b7-27f670e8ef16-image-registry-private-configuration\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.880087 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.880074 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2xn2\" (UniqueName: \"kubernetes.io/projected/94439f31-71d0-49e6-81b7-27f670e8ef16-kube-api-access-c2xn2\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.880168 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.880095 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94439f31-71d0-49e6-81b7-27f670e8ef16-registry-tls\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.880168 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.880118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94439f31-71d0-49e6-81b7-27f670e8ef16-bound-sa-token\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.880168 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.880151 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94439f31-71d0-49e6-81b7-27f670e8ef16-trusted-ca\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.980609 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.980527 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/94439f31-71d0-49e6-81b7-27f670e8ef16-registry-certificates\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.980609 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.980585 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/94439f31-71d0-49e6-81b7-27f670e8ef16-ca-trust-extracted\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.980832 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.980614 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/94439f31-71d0-49e6-81b7-27f670e8ef16-image-registry-private-configuration\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.980832 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.980641 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2xn2\" (UniqueName: \"kubernetes.io/projected/94439f31-71d0-49e6-81b7-27f670e8ef16-kube-api-access-c2xn2\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.980832 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.980690 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94439f31-71d0-49e6-81b7-27f670e8ef16-registry-tls\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.980832 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.980721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94439f31-71d0-49e6-81b7-27f670e8ef16-bound-sa-token\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.980832 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.980753 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94439f31-71d0-49e6-81b7-27f670e8ef16-trusted-ca\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.980832 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.980788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/94439f31-71d0-49e6-81b7-27f670e8ef16-installation-pull-secrets\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.981119 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.981086 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/94439f31-71d0-49e6-81b7-27f670e8ef16-ca-trust-extracted\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.981557 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.981529 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/94439f31-71d0-49e6-81b7-27f670e8ef16-registry-certificates\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.981728 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.981707 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94439f31-71d0-49e6-81b7-27f670e8ef16-trusted-ca\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.983199 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.983172 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/94439f31-71d0-49e6-81b7-27f670e8ef16-installation-pull-secrets\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.983293 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.983260 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94439f31-71d0-49e6-81b7-27f670e8ef16-registry-tls\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.983408 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.983388 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/94439f31-71d0-49e6-81b7-27f670e8ef16-image-registry-private-configuration\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.991535 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.991510 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94439f31-71d0-49e6-81b7-27f670e8ef16-bound-sa-token\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:01.991629 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:01.991590 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2xn2\" (UniqueName: \"kubernetes.io/projected/94439f31-71d0-49e6-81b7-27f670e8ef16-kube-api-access-c2xn2\") pod \"image-registry-5fb7478855-8rd8t\" (UID: \"94439f31-71d0-49e6-81b7-27f670e8ef16\") " pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:02.064047 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.064013 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-skqd4\"" Jul 10 18:01:02.072169 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.072146 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:02.202442 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.202409 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fb7478855-8rd8t"] Jul 10 18:01:02.205109 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:01:02.205082 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94439f31_71d0_49e6_81b7_27f670e8ef16.slice/crio-9f501c4d42c4487bbfc3991258c705abdfe7a6ccc434ad4552b09f22b64d8de2 WatchSource:0}: Error finding container 9f501c4d42c4487bbfc3991258c705abdfe7a6ccc434ad4552b09f22b64d8de2: Status 404 returned error can't find the container with id 9f501c4d42c4487bbfc3991258c705abdfe7a6ccc434ad4552b09f22b64d8de2 Jul 10 18:01:02.278959 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.278924 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" event={"ID":"94439f31-71d0-49e6-81b7-27f670e8ef16","Type":"ContainerStarted","Data":"a619fcefd8f992195b28cca0486784d3139f32c27bd244f3446f552096cb19c5"} Jul 10 18:01:02.279153 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.278939 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:01:02.279153 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.278985 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" event={"ID":"94439f31-71d0-49e6-81b7-27f670e8ef16","Type":"ContainerStarted","Data":"9f501c4d42c4487bbfc3991258c705abdfe7a6ccc434ad4552b09f22b64d8de2"} Jul 10 18:01:02.279153 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.279101 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:02.284386 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.284365 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:01:02.300777 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.300728 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" podStartSLOduration=1.300712069 podStartE2EDuration="1.300712069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:02.299933593 +0000 UTC m=+65.846396739" watchObservedRunningTime="2026-07-10 18:01:02.300712069 +0000 UTC m=+65.847175211" Jul 10 18:01:02.384188 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.384149 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/236ea491-caa7-41b3-9bf6-ca03364826c8-ca-trust-extracted\") pod \"236ea491-caa7-41b3-9bf6-ca03364826c8\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " Jul 10 18:01:02.384360 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.384212 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-bound-sa-token\") pod \"236ea491-caa7-41b3-9bf6-ca03364826c8\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " Jul 10 18:01:02.384360 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.384253 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pptz7\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-kube-api-access-pptz7\") pod \"236ea491-caa7-41b3-9bf6-ca03364826c8\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " Jul 10 18:01:02.384360 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.384272 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/236ea491-caa7-41b3-9bf6-ca03364826c8-trusted-ca\") pod \"236ea491-caa7-41b3-9bf6-ca03364826c8\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " Jul 10 18:01:02.384360 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.384290 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/236ea491-caa7-41b3-9bf6-ca03364826c8-image-registry-private-configuration\") pod \"236ea491-caa7-41b3-9bf6-ca03364826c8\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " Jul 10 18:01:02.384360 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.384311 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-certificates\") pod \"236ea491-caa7-41b3-9bf6-ca03364826c8\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " Jul 10 18:01:02.384360 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.384344 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/236ea491-caa7-41b3-9bf6-ca03364826c8-installation-pull-secrets\") pod \"236ea491-caa7-41b3-9bf6-ca03364826c8\" (UID: \"236ea491-caa7-41b3-9bf6-ca03364826c8\") " Jul 10 18:01:02.384655 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.384514 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/236ea491-caa7-41b3-9bf6-ca03364826c8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "236ea491-caa7-41b3-9bf6-ca03364826c8" (UID: "236ea491-caa7-41b3-9bf6-ca03364826c8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:01:02.384796 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.384766 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/236ea491-caa7-41b3-9bf6-ca03364826c8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "236ea491-caa7-41b3-9bf6-ca03364826c8" (UID: "236ea491-caa7-41b3-9bf6-ca03364826c8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:01:02.384874 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.384775 2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/236ea491-caa7-41b3-9bf6-ca03364826c8-ca-trust-extracted\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:01:02.385425 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.385387 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "236ea491-caa7-41b3-9bf6-ca03364826c8" (UID: "236ea491-caa7-41b3-9bf6-ca03364826c8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:01:02.387055 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.387031 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/236ea491-caa7-41b3-9bf6-ca03364826c8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "236ea491-caa7-41b3-9bf6-ca03364826c8" (UID: "236ea491-caa7-41b3-9bf6-ca03364826c8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:01:02.387153 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.387053 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/236ea491-caa7-41b3-9bf6-ca03364826c8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "236ea491-caa7-41b3-9bf6-ca03364826c8" (UID: "236ea491-caa7-41b3-9bf6-ca03364826c8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:01:02.387218 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.387159 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-kube-api-access-pptz7" (OuterVolumeSpecName: "kube-api-access-pptz7") pod "236ea491-caa7-41b3-9bf6-ca03364826c8" (UID: "236ea491-caa7-41b3-9bf6-ca03364826c8"). InnerVolumeSpecName "kube-api-access-pptz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:01:02.387408 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.387391 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "236ea491-caa7-41b3-9bf6-ca03364826c8" (UID: "236ea491-caa7-41b3-9bf6-ca03364826c8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:01:02.485751 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.485654 2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/236ea491-caa7-41b3-9bf6-ca03364826c8-image-registry-private-configuration\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:01:02.485751 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.485701 2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-certificates\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:01:02.485751 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.485715 2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/236ea491-caa7-41b3-9bf6-ca03364826c8-installation-pull-secrets\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:01:02.485751 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.485730 2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-bound-sa-token\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:01:02.485751 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.485743 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pptz7\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-kube-api-access-pptz7\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:01:02.485751 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.485756 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/236ea491-caa7-41b3-9bf6-ca03364826c8-trusted-ca\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:01:02.688307 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.688267 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs\") pod \"network-metrics-daemon-z5g5q\" (UID: \"c1121149-7010-4fb2-a378-dc79d124db58\") " pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:01:02.690956 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.690932 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 18:01:02.701296 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.701271 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1121149-7010-4fb2-a378-dc79d124db58-metrics-certs\") pod \"network-metrics-daemon-z5g5q\" (UID: \"c1121149-7010-4fb2-a378-dc79d124db58\") " pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:01:02.789192 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.789080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:01:02.791888 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.791864 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 18:01:02.802093 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.802060 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12a0c5b8-e1de-44de-a231-5a56fa8a1d23-original-pull-secret\") pod \"global-pull-secret-syncer-g8xxt\" (UID: \"12a0c5b8-e1de-44de-a231-5a56fa8a1d23\") " pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:01:02.890521 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.890484 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhqk\" (UniqueName: \"kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk\") pod \"network-check-target-z7dtl\" (UID: \"9766e246-9e80-449e-b500-e290fa4ac5c1\") " pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:01:02.892867 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.892847 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhqk\" (UniqueName: \"kubernetes.io/projected/9766e246-9e80-449e-b500-e290fa4ac5c1-kube-api-access-vbhqk\") pod \"network-check-target-z7dtl\" (UID: \"9766e246-9e80-449e-b500-e290fa4ac5c1\") " pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:01:02.994133 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.994114 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4tg5\"" Jul 10 18:01:02.999889 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:02.999873 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-25vgp\"" Jul 10 18:01:03.000862 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:03.000847 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5g5q" Jul 10 18:01:03.003558 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:03.003530 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8xxt" Jul 10 18:01:03.008219 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:03.008202 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:01:03.141527 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:03.141486 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z5g5q"] Jul 10 18:01:03.145236 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:01:03.145208 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1121149_7010_4fb2_a378_dc79d124db58.slice/crio-726be5e2649b4b977cdea14770c6825d80cf12cd7dcc92b8d9ea4c16a0806aa6 WatchSource:0}: Error finding container 726be5e2649b4b977cdea14770c6825d80cf12cd7dcc92b8d9ea4c16a0806aa6: Status 404 returned error can't find the container with id 726be5e2649b4b977cdea14770c6825d80cf12cd7dcc92b8d9ea4c16a0806aa6 Jul 10 18:01:03.282741 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:03.282717 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbbfdc745-jjjn6" Jul 10 18:01:03.282929 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:03.282707 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z5g5q" event={"ID":"c1121149-7010-4fb2-a378-dc79d124db58","Type":"ContainerStarted","Data":"726be5e2649b4b977cdea14770c6825d80cf12cd7dcc92b8d9ea4c16a0806aa6"} Jul 10 18:01:03.311498 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:03.311409 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bbbfdc745-jjjn6"] Jul 10 18:01:03.316623 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:03.316596 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bbbfdc745-jjjn6"] Jul 10 18:01:03.367379 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:03.367348 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g8xxt"] Jul 10 18:01:03.370413 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:03.370385 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7dtl"] Jul 10 18:01:03.371355 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:01:03.371267 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a0c5b8_e1de_44de_a231_5a56fa8a1d23.slice/crio-0ea5ea440773b2d6bba6cb18c93a43647f86f59a250348cb3cf9d7da1c7eb09c WatchSource:0}: Error finding container 0ea5ea440773b2d6bba6cb18c93a43647f86f59a250348cb3cf9d7da1c7eb09c: Status 404 returned error can't find the container with id 0ea5ea440773b2d6bba6cb18c93a43647f86f59a250348cb3cf9d7da1c7eb09c Jul 10 18:01:03.373587 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:01:03.373522 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9766e246_9e80_449e_b500_e290fa4ac5c1.slice/crio-08ac68d5276a6160a6f3ab95be3047dceddcbd24a70f5ade7808ec93be4cbebd WatchSource:0}: Error finding container 08ac68d5276a6160a6f3ab95be3047dceddcbd24a70f5ade7808ec93be4cbebd: Status 404 returned error can't find the container with id 08ac68d5276a6160a6f3ab95be3047dceddcbd24a70f5ade7808ec93be4cbebd Jul 10 18:01:03.394514 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:03.394488 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/236ea491-caa7-41b3-9bf6-ca03364826c8-registry-tls\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:01:04.152802 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.152775 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:01:04.153200 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.152845 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:01:04.153256 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.153217 2569 scope.go:117] "RemoveContainer" containerID="60abe0ba19a1a0a16b4efdecec1fe04c61a29f2189d19f7709010607965d094c" Jul 10 18:01:04.153736 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:01:04.153399 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-x72w7_openshift-console-operator(a4f78d82-214c-4612-adc5-32497c685a36)\"" pod="openshift-console-operator/console-operator-7866497494-x72w7" podUID="a4f78d82-214c-4612-adc5-32497c685a36" Jul 10 18:01:04.290071 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.290033 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z5g5q" event={"ID":"c1121149-7010-4fb2-a378-dc79d124db58","Type":"ContainerStarted","Data":"39631fd6f33ea361947145e86bca4f8a313fdd86505d4d4574e632887e3adb1c"} Jul 10 18:01:04.290256 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.290103 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z5g5q" event={"ID":"c1121149-7010-4fb2-a378-dc79d124db58","Type":"ContainerStarted","Data":"e2bf98311d76278dd4b6072bd605b7267e217baafb519208a49fba2cf1a408bb"} Jul 10 18:01:04.292102 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.292072 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7dtl" event={"ID":"9766e246-9e80-449e-b500-e290fa4ac5c1","Type":"ContainerStarted","Data":"c70499db8a1ca0498972daa3fc5aa94cb7978314c615ed21ab071cc22c83cbb1"} Jul 10 18:01:04.292219 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.292126 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7dtl" event={"ID":"9766e246-9e80-449e-b500-e290fa4ac5c1","Type":"ContainerStarted","Data":"08ac68d5276a6160a6f3ab95be3047dceddcbd24a70f5ade7808ec93be4cbebd"} Jul 10 18:01:04.292219 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.292148 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:01:04.293381 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.293353 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g8xxt" event={"ID":"12a0c5b8-e1de-44de-a231-5a56fa8a1d23","Type":"ContainerStarted","Data":"0ea5ea440773b2d6bba6cb18c93a43647f86f59a250348cb3cf9d7da1c7eb09c"} Jul 10 18:01:04.303333 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.303308 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:01:04.305738 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.305712 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6626849-dd7f-41ff-b019-04d42ae9229d-metrics-tls\") pod \"dns-default-t6b2r\" (UID: \"b6626849-dd7f-41ff-b019-04d42ae9229d\") " pod="openshift-dns/dns-default-t6b2r" Jul 10 18:01:04.308282 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.308242 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z5g5q" podStartSLOduration=66.387522834 podStartE2EDuration="1m7.308228001s" totalImagesPullingTime="920.705167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:04.307239615 +0000 UTC m=+67.853702790" watchObservedRunningTime="2026-07-10 18:01:04.308228001 +0000 UTC m=+67.854691142" Jul 10 18:01:04.322094 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.322008 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z7dtl" podStartSLOduration=67.321992841 podStartE2EDuration="1m7.321992841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:04.321519564 +0000 UTC m=+67.867982732" watchObservedRunningTime="2026-07-10 18:01:04.321992841 +0000 UTC m=+67.868455983" Jul 10 18:01:04.464389 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.464351 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xxlp9\"" Jul 10 18:01:04.472244 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.472214 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t6b2r" Jul 10 18:01:04.615949 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.615761 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t6b2r"] Jul 10 18:01:04.618904 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:01:04.618865 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6626849_dd7f_41ff_b019_04d42ae9229d.slice/crio-dbd551a668a7bbc39a646fc4bea1442cc4c83e0b5a508f45bbde5ecd4eeaeece WatchSource:0}: Error finding container dbd551a668a7bbc39a646fc4bea1442cc4c83e0b5a508f45bbde5ecd4eeaeece: Status 404 returned error can't find the container with id dbd551a668a7bbc39a646fc4bea1442cc4c83e0b5a508f45bbde5ecd4eeaeece Jul 10 18:01:04.983442 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:04.983360 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="236ea491-caa7-41b3-9bf6-ca03364826c8" path="/var/lib/kubelet/pods/236ea491-caa7-41b3-9bf6-ca03364826c8/volumes" Jul 10 18:01:05.298662 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:05.298572 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t6b2r" event={"ID":"b6626849-dd7f-41ff-b019-04d42ae9229d","Type":"ContainerStarted","Data":"dbd551a668a7bbc39a646fc4bea1442cc4c83e0b5a508f45bbde5ecd4eeaeece"} Jul 10 18:01:05.818407 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:05.818364 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:01:05.818620 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:05.818430 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdrv\" (UID: \"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:01:05.818620 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:05.818469 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:01:05.819260 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:05.819231 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d16fa6-3622-410f-acac-d4ddfe4704b2-service-ca-bundle\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:01:05.821546 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:05.821488 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d16fa6-3622-410f-acac-d4ddfe4704b2-metrics-certs\") pod \"router-default-5b875d77bd-5c2fl\" (UID: \"14d16fa6-3622-410f-acac-d4ddfe4704b2\") " pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:01:05.821546 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:05.821495 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5e7d458-2d34-4dab-90f3-cd2f0d7327a3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-wtdrv\" (UID: \"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:01:05.964719 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:05.964677 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9kxw5\"" Jul 10 18:01:05.972749 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:05.972713 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" Jul 10 18:01:05.985558 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:05.985530 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-h58f6\"" Jul 10 18:01:05.993605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:05.993580 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:01:06.675638 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:06.675607 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv"] Jul 10 18:01:06.696194 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:06.696120 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b875d77bd-5c2fl"] Jul 10 18:01:06.703928 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:01:06.703902 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d16fa6_3622_410f_acac_d4ddfe4704b2.slice/crio-b33bf36dbfcb2afea095c72780c9958eeb2967a09295a62c3c38e2dbf1e75008 WatchSource:0}: Error finding container b33bf36dbfcb2afea095c72780c9958eeb2967a09295a62c3c38e2dbf1e75008: Status 404 returned error can't find the container with id b33bf36dbfcb2afea095c72780c9958eeb2967a09295a62c3c38e2dbf1e75008 Jul 10 18:01:07.307333 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:07.307297 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" event={"ID":"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3","Type":"ContainerStarted","Data":"eeff594aaa967d5cad15c5a404e78824eda9d107ead564bf680270de1dc356c7"} Jul 10 18:01:07.308473 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:07.308447 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b875d77bd-5c2fl" event={"ID":"14d16fa6-3622-410f-acac-d4ddfe4704b2","Type":"ContainerStarted","Data":"b33bf36dbfcb2afea095c72780c9958eeb2967a09295a62c3c38e2dbf1e75008"} Jul 10 18:01:08.698390 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:08.698358 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bvh74"] Jul 10 18:01:08.700962 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:08.700941 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bvh74" Jul 10 18:01:08.704485 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:08.704100 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 18:01:08.704485 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:08.704361 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-c2ncf\"" Jul 10 18:01:08.712007 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:08.711982 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bvh74"] Jul 10 18:01:08.848086 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:08.848049 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2eb92c4e-4ea0-427d-82bd-ccdc82d2b2b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-bvh74\" (UID: \"2eb92c4e-4ea0-427d-82bd-ccdc82d2b2b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bvh74" Jul 10 18:01:08.948916 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:08.948840 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2eb92c4e-4ea0-427d-82bd-ccdc82d2b2b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-bvh74\" (UID: \"2eb92c4e-4ea0-427d-82bd-ccdc82d2b2b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bvh74" Jul 10 18:01:08.951686 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:08.951654 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2eb92c4e-4ea0-427d-82bd-ccdc82d2b2b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-bvh74\" (UID: \"2eb92c4e-4ea0-427d-82bd-ccdc82d2b2b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bvh74" Jul 10 18:01:09.013865 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:09.013836 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bvh74" Jul 10 18:01:09.143555 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:09.143528 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bvh74"] Jul 10 18:01:09.147081 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:01:09.147053 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb92c4e_4ea0_427d_82bd_ccdc82d2b2b9.slice/crio-5a4959a41630dfc4ceddc8a04408df2e458e9d368482f6018b59d297f9b56170 WatchSource:0}: Error finding container 5a4959a41630dfc4ceddc8a04408df2e458e9d368482f6018b59d297f9b56170: Status 404 returned error can't find the container with id 5a4959a41630dfc4ceddc8a04408df2e458e9d368482f6018b59d297f9b56170 Jul 10 18:01:09.317695 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:09.317653 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t6b2r" event={"ID":"b6626849-dd7f-41ff-b019-04d42ae9229d","Type":"ContainerStarted","Data":"2a03d0741b7b394ee155d54fa134f53da9361ac682ef20893d326f2cd084b746"} Jul 10 18:01:09.317891 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:09.317724 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t6b2r" event={"ID":"b6626849-dd7f-41ff-b019-04d42ae9229d","Type":"ContainerStarted","Data":"9c0c179282aba3c3652f1bd65fbe29fffb15b4e0fdeef842201c5a47ad8b3374"} Jul 10 18:01:09.317891 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:09.317851 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t6b2r" Jul 10 18:01:09.319311 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:09.319274 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b875d77bd-5c2fl" event={"ID":"14d16fa6-3622-410f-acac-d4ddfe4704b2","Type":"ContainerStarted","Data":"9e4b3fca200784f331c31f7f92794c62d9a19da14044741b1608d0a5833e89f3"} Jul 10 18:01:09.320423 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:09.320402 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bvh74" event={"ID":"2eb92c4e-4ea0-427d-82bd-ccdc82d2b2b9","Type":"ContainerStarted","Data":"5a4959a41630dfc4ceddc8a04408df2e458e9d368482f6018b59d297f9b56170"} Jul 10 18:01:09.322077 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:09.321791 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g8xxt" event={"ID":"12a0c5b8-e1de-44de-a231-5a56fa8a1d23","Type":"ContainerStarted","Data":"e903ee5285e401a127e448affa1b20877c6d6488a1d8cbe756b8fe420326d89d"} Jul 10 18:01:09.336395 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:09.336345 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t6b2r" podStartSLOduration=33.695575448 podStartE2EDuration="37.336327795s" totalImagesPullingTime="3.640752347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:09.334107651 +0000 UTC m=+72.880570814" watchObservedRunningTime="2026-07-10 18:01:09.336327795 +0000 UTC m=+72.882790960" Jul 10 18:01:09.348985 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:09.348945 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g8xxt" podStartSLOduration=67.425810964 podStartE2EDuration="1m12.348934033s" totalImagesPullingTime="4.923123069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:09.348250708 +0000 UTC m=+72.894713851" watchObservedRunningTime="2026-07-10 18:01:09.348934033 +0000 UTC m=+72.895397174" Jul 10 18:01:09.382548 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:09.382498 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5b875d77bd-5c2fl" podStartSLOduration=36.382484858 podStartE2EDuration="36.382484858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:09.38225669 +0000 UTC m=+72.928719834" watchObservedRunningTime="2026-07-10 18:01:09.382484858 +0000 UTC m=+72.928948004" Jul 10 18:01:09.994190 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:09.994152 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:01:09.997064 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:09.997039 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:01:10.327478 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:10.327438 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" event={"ID":"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3","Type":"ContainerStarted","Data":"67305d67b830298f06ee7ce93ec8838803f21014c361349a302e4037eba4b583"} Jul 10 18:01:10.327659 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:10.327495 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" event={"ID":"a5e7d458-2d34-4dab-90f3-cd2f0d7327a3","Type":"ContainerStarted","Data":"6bf9fd99f73e9aef7a5fb20f560f8f4979b6e878ebb043820afe1b5cc6abfbea"} Jul 10 18:01:10.328014 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:10.327989 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:01:10.329180 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:10.329160 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5b875d77bd-5c2fl" Jul 10 18:01:10.361410 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:10.361360 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-wtdrv" podStartSLOduration=34.456255146 podStartE2EDuration="37.361338031s" totalImagesPullingTime="2.905082885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:10.359598667 +0000 UTC m=+73.906061809" watchObservedRunningTime="2026-07-10 18:01:10.361338031 +0000 UTC m=+73.907801173" Jul 10 18:01:11.335638 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.335595 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bvh74" event={"ID":"2eb92c4e-4ea0-427d-82bd-ccdc82d2b2b9","Type":"ContainerStarted","Data":"c519c0189b1203c0593263d770730fbacfc4d1340994467fc4388b19579acc73"} Jul 10 18:01:11.336077 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.335784 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bvh74" Jul 10 18:01:11.340645 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.340616 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bvh74" Jul 10 18:01:11.349959 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.349920 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bvh74" podStartSLOduration=1.91721951 podStartE2EDuration="3.349907626s" totalImagesPullingTime="1.432688116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:11.349651557 +0000 UTC m=+74.896114699" watchObservedRunningTime="2026-07-10 18:01:11.349907626 +0000 UTC m=+74.896370767" Jul 10 18:01:11.752979 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.752906 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-kgnvb"] Jul 10 18:01:11.754893 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.754878 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:11.757275 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.757246 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 10 18:01:11.757388 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.757250 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 10 18:01:11.757388 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.757299 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 18:01:11.758292 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.758275 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 18:01:11.758353 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.758275 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 18:01:11.758353 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.758275 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4lgdk\"" Jul 10 18:01:11.766453 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.766432 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-kgnvb"] Jul 10 18:01:11.874568 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.874542 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8949v\" (UniqueName: \"kubernetes.io/projected/91c7ad09-d763-4204-8771-8324e0b5d0cb-kube-api-access-8949v\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:11.874691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.874585 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c7ad09-d763-4204-8771-8324e0b5d0cb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:11.874691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.874605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91c7ad09-d763-4204-8771-8324e0b5d0cb-metrics-client-ca\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:11.874691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.874684 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91c7ad09-d763-4204-8771-8324e0b5d0cb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:11.975733 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.975704 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8949v\" (UniqueName: \"kubernetes.io/projected/91c7ad09-d763-4204-8771-8324e0b5d0cb-kube-api-access-8949v\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:11.975874 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.975762 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c7ad09-d763-4204-8771-8324e0b5d0cb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:11.975874 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.975788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91c7ad09-d763-4204-8771-8324e0b5d0cb-metrics-client-ca\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:11.975874 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.975860 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91c7ad09-d763-4204-8771-8324e0b5d0cb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:11.976038 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:01:11.975910 2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 10 18:01:11.976038 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:01:11.975966 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c7ad09-d763-4204-8771-8324e0b5d0cb-prometheus-operator-tls podName:91c7ad09-d763-4204-8771-8324e0b5d0cb nodeName:}" failed. No retries permitted until 2026-07-10 18:01:12.475951882 +0000 UTC m=+76.022415002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/91c7ad09-d763-4204-8771-8324e0b5d0cb-prometheus-operator-tls") pod "prometheus-operator-64785b6675-kgnvb" (UID: "91c7ad09-d763-4204-8771-8324e0b5d0cb") : secret "prometheus-operator-tls" not found Jul 10 18:01:11.976413 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.976393 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91c7ad09-d763-4204-8771-8324e0b5d0cb-metrics-client-ca\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:11.978265 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.978244 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91c7ad09-d763-4204-8771-8324e0b5d0cb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:11.985940 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:11.985918 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8949v\" (UniqueName: \"kubernetes.io/projected/91c7ad09-d763-4204-8771-8324e0b5d0cb-kube-api-access-8949v\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:12.479516 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:12.479463 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c7ad09-d763-4204-8771-8324e0b5d0cb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:12.479985 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:01:12.479665 2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 10 18:01:12.479985 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:01:12.479745 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c7ad09-d763-4204-8771-8324e0b5d0cb-prometheus-operator-tls podName:91c7ad09-d763-4204-8771-8324e0b5d0cb nodeName:}" failed. No retries permitted until 2026-07-10 18:01:13.479720578 +0000 UTC m=+77.026183721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/91c7ad09-d763-4204-8771-8324e0b5d0cb-prometheus-operator-tls") pod "prometheus-operator-64785b6675-kgnvb" (UID: "91c7ad09-d763-4204-8771-8324e0b5d0cb") : secret "prometheus-operator-tls" not found Jul 10 18:01:13.486562 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:13.486525 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c7ad09-d763-4204-8771-8324e0b5d0cb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:13.488852 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:13.488829 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c7ad09-d763-4204-8771-8324e0b5d0cb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-kgnvb\" (UID: \"91c7ad09-d763-4204-8771-8324e0b5d0cb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:13.563781 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:13.563748 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" Jul 10 18:01:13.698125 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:13.698094 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-kgnvb"] Jul 10 18:01:13.700896 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:01:13.700871 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c7ad09_d763_4204_8771_8324e0b5d0cb.slice/crio-47fce6a4ac634e96825da69a717ad470d6f2f7054f224a2c7653f71f26153f72 WatchSource:0}: Error finding container 47fce6a4ac634e96825da69a717ad470d6f2f7054f224a2c7653f71f26153f72: Status 404 returned error can't find the container with id 47fce6a4ac634e96825da69a717ad470d6f2f7054f224a2c7653f71f26153f72 Jul 10 18:01:14.349637 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:14.349599 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" event={"ID":"91c7ad09-d763-4204-8771-8324e0b5d0cb","Type":"ContainerStarted","Data":"47fce6a4ac634e96825da69a717ad470d6f2f7054f224a2c7653f71f26153f72"} Jul 10 18:01:14.895778 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:14.895747 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:01:14.898171 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:14.898144 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d86p4\" (UID: \"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca\") " pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:01:15.145741 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:15.145716 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pcwng\"" Jul 10 18:01:15.154630 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:15.154564 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d86p4" Jul 10 18:01:15.349683 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:15.349655 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d86p4"] Jul 10 18:01:15.352851 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:01:15.352825 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea76ed28_d4d8_49ef_b8b4_4c5a7f9c38ca.slice/crio-cc58f4ac25546b02456c826c8ca0a99c606c0d2cc02dae24323c688e83c475d8 WatchSource:0}: Error finding container cc58f4ac25546b02456c826c8ca0a99c606c0d2cc02dae24323c688e83c475d8: Status 404 returned error can't find the container with id cc58f4ac25546b02456c826c8ca0a99c606c0d2cc02dae24323c688e83c475d8 Jul 10 18:01:16.358469 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:16.358348 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d86p4" event={"ID":"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca","Type":"ContainerStarted","Data":"cc58f4ac25546b02456c826c8ca0a99c606c0d2cc02dae24323c688e83c475d8"} Jul 10 18:01:16.360476 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:16.360443 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" event={"ID":"91c7ad09-d763-4204-8771-8324e0b5d0cb","Type":"ContainerStarted","Data":"5bbc5b32d2b94e859162a71163aad7b39c6166cda52adcaba2de6bb35d9a54f1"} Jul 10 18:01:16.360608 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:16.360495 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" event={"ID":"91c7ad09-d763-4204-8771-8324e0b5d0cb","Type":"ContainerStarted","Data":"3fa79934954061e36831f42009cdbe02c2abc861e067bf0f4906210cb5f684cb"} Jul 10 18:01:16.980149 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:16.980123 2569 scope.go:117] "RemoveContainer" containerID="60abe0ba19a1a0a16b4efdecec1fe04c61a29f2189d19f7709010607965d094c" Jul 10 18:01:16.980300 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:01:16.980282 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-x72w7_openshift-console-operator(a4f78d82-214c-4612-adc5-32497c685a36)\"" pod="openshift-console-operator/console-operator-7866497494-x72w7" podUID="a4f78d82-214c-4612-adc5-32497c685a36" Jul 10 18:01:18.116595 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.116542 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-kgnvb" podStartSLOduration=5.224790621 podStartE2EDuration="7.116522445s" totalImagesPullingTime="1.891731824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:01:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:16.376445928 +0000 UTC m=+79.922909070" watchObservedRunningTime="2026-07-10 18:01:18.116522445 +0000 UTC m=+81.662985586" Jul 10 18:01:18.116982 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.116828 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-x4vcg"] Jul 10 18:01:18.121838 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.121801 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.124319 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.124299 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 10 18:01:18.124419 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.124348 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 10 18:01:18.125423 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.125365 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 10 18:01:18.125545 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.125498 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-48kbl\"" Jul 10 18:01:18.126949 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.126928 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8lg4v"] Jul 10 18:01:18.131178 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.131157 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.131422 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.131338 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-x4vcg"] Jul 10 18:01:18.135179 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.135159 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 18:01:18.135864 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.135845 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 18:01:18.135946 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.135850 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 18:01:18.136010 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.135971 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sqvhc\"" Jul 10 18:01:18.222532 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222499 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-tls\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.222532 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222536 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-textfile\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.222767 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222555 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c91b6298-ea96-4042-8fc8-7145c6c87aa6-metrics-client-ca\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.222767 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222582 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.222767 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222665 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c91b6298-ea96-4042-8fc8-7145c6c87aa6-sys\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.222767 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222697 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.222767 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222717 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.222767 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222742 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c91b6298-ea96-4042-8fc8-7145c6c87aa6-root\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.222767 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222759 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrdc\" (UniqueName: \"kubernetes.io/projected/c91b6298-ea96-4042-8fc8-7145c6c87aa6-kube-api-access-xbrdc\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.223140 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222785 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.223140 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222891 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5sqb\" (UniqueName: \"kubernetes.io/projected/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-api-access-w5sqb\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.223140 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222910 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-wtmp\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.223140 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222938 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a25093f-9aee-4ca0-8996-08949e85bcf7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.223140 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.222956 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5a25093f-9aee-4ca0-8996-08949e85bcf7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.223140 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.223025 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.324418 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324376 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5sqb\" (UniqueName: \"kubernetes.io/projected/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-api-access-w5sqb\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.324588 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324422 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-wtmp\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.324588 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324471 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a25093f-9aee-4ca0-8996-08949e85bcf7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.324588 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324501 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5a25093f-9aee-4ca0-8996-08949e85bcf7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.324588 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324548 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.324588 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324579 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-tls\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.324871 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324604 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-textfile\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.324871 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324624 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c91b6298-ea96-4042-8fc8-7145c6c87aa6-metrics-client-ca\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.324871 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324637 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-wtmp\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.324871 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324651 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.324871 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324748 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c91b6298-ea96-4042-8fc8-7145c6c87aa6-sys\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.324871 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.324871 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324832 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.324871 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324862 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c91b6298-ea96-4042-8fc8-7145c6c87aa6-root\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.325238 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324885 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrdc\" (UniqueName: \"kubernetes.io/projected/c91b6298-ea96-4042-8fc8-7145c6c87aa6-kube-api-access-xbrdc\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.325238 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324897 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c91b6298-ea96-4042-8fc8-7145c6c87aa6-sys\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.325238 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.324914 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.325238 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:01:18.324928 2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 18:01:18.325238 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:01:18.325001 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-tls podName:c91b6298-ea96-4042-8fc8-7145c6c87aa6 nodeName:}" failed. No retries permitted until 2026-07-10 18:01:18.824978428 +0000 UTC m=+82.371441562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-tls") pod "node-exporter-8lg4v" (UID: "c91b6298-ea96-4042-8fc8-7145c6c87aa6") : secret "node-exporter-tls" not found Jul 10 18:01:18.325238 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:01:18.325029 2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 10 18:01:18.325238 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:01:18.325090 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-state-metrics-tls podName:5a25093f-9aee-4ca0-8996-08949e85bcf7 nodeName:}" failed. No retries permitted until 2026-07-10 18:01:18.825071255 +0000 UTC m=+82.371534376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-x4vcg" (UID: "5a25093f-9aee-4ca0-8996-08949e85bcf7") : secret "kube-state-metrics-tls" not found Jul 10 18:01:18.325238 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.325121 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5a25093f-9aee-4ca0-8996-08949e85bcf7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.325238 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.325224 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c91b6298-ea96-4042-8fc8-7145c6c87aa6-root\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.325675 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.325411 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-textfile\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.325675 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.325488 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c91b6298-ea96-4042-8fc8-7145c6c87aa6-metrics-client-ca\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.325675 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.325630 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.326121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.326100 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a25093f-9aee-4ca0-8996-08949e85bcf7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.326261 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.326126 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.327415 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.327390 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.327625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.327609 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.334056 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.334013 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrdc\" (UniqueName: \"kubernetes.io/projected/c91b6298-ea96-4042-8fc8-7145c6c87aa6-kube-api-access-xbrdc\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.334159 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.334089 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5sqb\" (UniqueName: \"kubernetes.io/projected/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-api-access-w5sqb\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.830127 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.830096 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.830413 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.830178 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-tls\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:18.834423 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.833386 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a25093f-9aee-4ca0-8996-08949e85bcf7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-x4vcg\" (UID: \"5a25093f-9aee-4ca0-8996-08949e85bcf7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:18.835828 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:18.835790 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c91b6298-ea96-4042-8fc8-7145c6c87aa6-node-exporter-tls\") pod \"node-exporter-8lg4v\" (UID: \"c91b6298-ea96-4042-8fc8-7145c6c87aa6\") " pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:19.032947 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:19.032919 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" Jul 10 18:01:19.040904 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:19.040880 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8lg4v" Jul 10 18:01:19.060218 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:01:19.060193 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91b6298_ea96_4042_8fc8_7145c6c87aa6.slice/crio-c3885353fb88de5ac7be81b4f9ce631938588d402ce5bf05f0b395c562ae1afb WatchSource:0}: Error finding container c3885353fb88de5ac7be81b4f9ce631938588d402ce5bf05f0b395c562ae1afb: Status 404 returned error can't find the container with id c3885353fb88de5ac7be81b4f9ce631938588d402ce5bf05f0b395c562ae1afb Jul 10 18:01:19.191925 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:19.191900 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-x4vcg"] Jul 10 18:01:19.193999 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:01:19.193959 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a25093f_9aee_4ca0_8996_08949e85bcf7.slice/crio-d3461f1a8fadecf4ca601fe3bc6bf76ff3c6015f806caea852caee1b24767007 WatchSource:0}: Error finding container d3461f1a8fadecf4ca601fe3bc6bf76ff3c6015f806caea852caee1b24767007: Status 404 returned error can't find the container with id d3461f1a8fadecf4ca601fe3bc6bf76ff3c6015f806caea852caee1b24767007 Jul 10 18:01:19.330306 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:19.330274 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t6b2r" Jul 10 18:01:19.373539 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:19.373466 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d86p4" event={"ID":"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca","Type":"ContainerStarted","Data":"a3e1a7b28b6bc6986e009e24b9a1cb8b27052675dd408c4d52cbd471f3d1b0a3"} Jul 10 18:01:19.374864 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:19.374835 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" event={"ID":"5a25093f-9aee-4ca0-8996-08949e85bcf7","Type":"ContainerStarted","Data":"d3461f1a8fadecf4ca601fe3bc6bf76ff3c6015f806caea852caee1b24767007"} Jul 10 18:01:19.375931 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:19.375898 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lg4v" event={"ID":"c91b6298-ea96-4042-8fc8-7145c6c87aa6","Type":"ContainerStarted","Data":"c3885353fb88de5ac7be81b4f9ce631938588d402ce5bf05f0b395c562ae1afb"} Jul 10 18:01:20.382699 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:20.382660 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d86p4" event={"ID":"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca","Type":"ContainerStarted","Data":"7d63b73d3b8759c0089f0fcac855aca72589a1661f99d3abfcc1df06244f319c"} Jul 10 18:01:21.389130 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:21.389092 2569 generic.go:356] "Generic (PLEG): container finished" podID="c91b6298-ea96-4042-8fc8-7145c6c87aa6" containerID="3ac513f2c934526dc6a9b90fc2e5aab9cc2cda6eac9a3577910c3a7e89fabfb3" exitCode=0 Jul 10 18:01:21.389667 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:21.389196 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lg4v" event={"ID":"c91b6298-ea96-4042-8fc8-7145c6c87aa6","Type":"ContainerDied","Data":"3ac513f2c934526dc6a9b90fc2e5aab9cc2cda6eac9a3577910c3a7e89fabfb3"} Jul 10 18:01:21.392063 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:21.392021 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" event={"ID":"5a25093f-9aee-4ca0-8996-08949e85bcf7","Type":"ContainerStarted","Data":"140c166243ec476872004c56c40aaf76627e733734e3e2af41fd5d7630b5c196"} Jul 10 18:01:21.392178 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:21.392080 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" event={"ID":"5a25093f-9aee-4ca0-8996-08949e85bcf7","Type":"ContainerStarted","Data":"f32d1c5224afb97e9423da98047369b6c5f7cdde9a76bdb1c262b1ceeeb9d92b"} Jul 10 18:01:21.392178 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:21.392114 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" event={"ID":"5a25093f-9aee-4ca0-8996-08949e85bcf7","Type":"ContainerStarted","Data":"453e99f2de8171d77a40bf3e2597a70db3d53398f07164dfec59329cd9b5aa60"} Jul 10 18:01:21.430914 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:21.430862 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-x4vcg" podStartSLOduration=2.151502968 podStartE2EDuration="3.430842581s" totalImagesPullingTime="1.279339613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:21.428233895 +0000 UTC m=+84.974697037" watchObservedRunningTime="2026-07-10 18:01:21.430842581 +0000 UTC m=+84.977305721" Jul 10 18:01:22.398695 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.398638 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lg4v" event={"ID":"c91b6298-ea96-4042-8fc8-7145c6c87aa6","Type":"ContainerStarted","Data":"732b6f2833ad43a08f1e5199fa9ee833a1ebf2c442db3884d9c06af517729d70"} Jul 10 18:01:22.399130 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.398722 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lg4v" event={"ID":"c91b6298-ea96-4042-8fc8-7145c6c87aa6","Type":"ContainerStarted","Data":"a7f0c3dba016bf38ba5646213d320eed493aee5e51d0e54d5577030d9b3725fc"} Jul 10 18:01:22.417959 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.417892 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8lg4v" podStartSLOduration=2.966136886 podStartE2EDuration="4.41787555s" totalImagesPullingTime="1.369598339s" totalInitContainerRuntime="82.140325ms" isStatefulPod=true podCreationTimestamp="2026-07-10 18:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:22.41640802 +0000 UTC m=+85.962871177" watchObservedRunningTime="2026-07-10 18:01:22.41787555 +0000 UTC m=+85.964338692" Jul 10 18:01:22.518138 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.518092 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8f8458798-2klml"] Jul 10 18:01:22.530031 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.529987 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.530465 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.530440 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8f8458798-2klml"] Jul 10 18:01:22.532975 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.532951 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 18:01:22.533125 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.533008 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 18:01:22.533195 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.533171 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 18:01:22.533242 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.533217 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 18:01:22.533459 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.533432 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4t5s76oil899j\"" Jul 10 18:01:22.533578 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.533557 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4ds24\"" Jul 10 18:01:22.663018 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.662940 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5680deaa-f23f-49c9-8aee-8995859cac4a-metrics-server-audit-profiles\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.663018 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.662984 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5680deaa-f23f-49c9-8aee-8995859cac4a-secret-metrics-server-tls\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.663018 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.663012 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5680deaa-f23f-49c9-8aee-8995859cac4a-client-ca-bundle\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.663253 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.663058 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5680deaa-f23f-49c9-8aee-8995859cac4a-secret-metrics-server-client-certs\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.663253 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.663081 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5680deaa-f23f-49c9-8aee-8995859cac4a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.663253 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.663115 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5680deaa-f23f-49c9-8aee-8995859cac4a-audit-log\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.663253 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.663144 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl2fg\" (UniqueName: \"kubernetes.io/projected/5680deaa-f23f-49c9-8aee-8995859cac4a-kube-api-access-wl2fg\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.763689 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.763657 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5680deaa-f23f-49c9-8aee-8995859cac4a-metrics-server-audit-profiles\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.763689 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.763693 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5680deaa-f23f-49c9-8aee-8995859cac4a-secret-metrics-server-tls\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.763938 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.763712 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5680deaa-f23f-49c9-8aee-8995859cac4a-client-ca-bundle\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.763938 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.763740 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5680deaa-f23f-49c9-8aee-8995859cac4a-secret-metrics-server-client-certs\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.763938 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.763772 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5680deaa-f23f-49c9-8aee-8995859cac4a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.763938 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.763830 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5680deaa-f23f-49c9-8aee-8995859cac4a-audit-log\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.763938 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.763862 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl2fg\" (UniqueName: \"kubernetes.io/projected/5680deaa-f23f-49c9-8aee-8995859cac4a-kube-api-access-wl2fg\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.764215 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.764191 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5680deaa-f23f-49c9-8aee-8995859cac4a-audit-log\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.764653 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.764629 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5680deaa-f23f-49c9-8aee-8995859cac4a-metrics-server-audit-profiles\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.764837 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.764794 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5680deaa-f23f-49c9-8aee-8995859cac4a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.766223 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.766200 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5680deaa-f23f-49c9-8aee-8995859cac4a-secret-metrics-server-tls\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.766317 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.766224 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5680deaa-f23f-49c9-8aee-8995859cac4a-client-ca-bundle\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.766522 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.766505 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5680deaa-f23f-49c9-8aee-8995859cac4a-secret-metrics-server-client-certs\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.771601 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.771579 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl2fg\" (UniqueName: \"kubernetes.io/projected/5680deaa-f23f-49c9-8aee-8995859cac4a-kube-api-access-wl2fg\") pod \"metrics-server-8f8458798-2klml\" (UID: \"5680deaa-f23f-49c9-8aee-8995859cac4a\") " pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.842188 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.842165 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:22.978359 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:22.978332 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8f8458798-2klml"] Jul 10 18:01:22.982360 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:01:22.982336 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5680deaa_f23f_49c9_8aee_8995859cac4a.slice/crio-816c09308c1dedafcbda3503ff40713b1ea964493ba6f048e47c29c72d32374b WatchSource:0}: Error finding container 816c09308c1dedafcbda3503ff40713b1ea964493ba6f048e47c29c72d32374b: Status 404 returned error can't find the container with id 816c09308c1dedafcbda3503ff40713b1ea964493ba6f048e47c29c72d32374b Jul 10 18:01:23.287356 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:23.287281 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fb7478855-8rd8t" Jul 10 18:01:23.404578 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:23.404542 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d86p4" event={"ID":"ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca","Type":"ContainerStarted","Data":"9e640b6786f91d0bdaccbb8cdea9baef9fdeb5ac6dd8f30a1bcb08c8b202656f"} Jul 10 18:01:23.405820 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:23.405787 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8f8458798-2klml" event={"ID":"5680deaa-f23f-49c9-8aee-8995859cac4a","Type":"ContainerStarted","Data":"816c09308c1dedafcbda3503ff40713b1ea964493ba6f048e47c29c72d32374b"} Jul 10 18:01:23.421560 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:23.421512 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d86p4" podStartSLOduration=34.059073604 podStartE2EDuration="41.421499788s" totalImagesPullingTime="7.362426184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:00:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:23.420507347 +0000 UTC m=+86.966970491" watchObservedRunningTime="2026-07-10 18:01:23.421499788 +0000 UTC m=+86.967962968" Jul 10 18:01:24.356492 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.355676 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 18:01:24.374338 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.374302 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.375346 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.375321 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 18:01:24.376745 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.376722 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 18:01:24.376745 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.376737 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 18:01:24.377020 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.376776 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 18:01:24.377020 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.376723 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 18:01:24.377020 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.376925 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 18:01:24.377226 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.377042 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-m6bp9\"" Jul 10 18:01:24.377671 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.377640 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9r9q93jf7jq0b\"" Jul 10 18:01:24.378198 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.377862 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 18:01:24.378198 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.378055 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 18:01:24.378445 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.378425 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 18:01:24.378550 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.378462 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 18:01:24.379133 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.378902 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 18:01:24.379133 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.379106 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 18:01:24.379292 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.379240 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 18:01:24.379292 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.379285 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 18:01:24.381901 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.381711 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.478818 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.478887 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.478914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.478953 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.478976 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479005 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-config\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479027 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479049 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479081 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-web-config\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479107 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479142 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479165 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479189 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwdn4\" (UniqueName: \"kubernetes.io/projected/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-kube-api-access-fwdn4\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479212 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.479605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479272 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.480610 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479307 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.480610 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479332 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.480610 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479358 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.480610 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.479387 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-config-out\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580385 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580432 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580457 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580480 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdn4\" (UniqueName: \"kubernetes.io/projected/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-kube-api-access-fwdn4\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580525 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580602 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580624 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580648 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580689 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-config-out\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580711 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580782 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580862 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580899 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.581691 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-config\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.582602 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580951 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.582602 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.580974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.582602 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.581005 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-web-config\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.583064 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.582931 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.583541 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.583509 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.583780 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.583756 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.584680 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.584653 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.585984 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.585222 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.585984 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.585628 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-web-config\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.585984 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.585761 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.586766 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.586737 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.587009 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.586989 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.587082 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.587006 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.587617 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.587593 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.587931 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.587875 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.588396 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.588369 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.588492 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.588407 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.589744 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.589528 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.589744 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.589693 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.589928 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.589760 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-config-out\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.590165 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.590119 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-config\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.590892 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.590867 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.592110 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.592091 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwdn4\" (UniqueName: \"kubernetes.io/projected/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-kube-api-access-fwdn4\") pod \"prometheus-k8s-0\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.688594 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.688563 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:24.824170 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:24.824145 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 18:01:24.827271 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:01:24.827235 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bea28d_d0a5_4898_a05f_bf2aa2628de7.slice/crio-d899153dfd77ff6146c829edc32c72d10639b1bba4b8ceec3626438d5472d141 WatchSource:0}: Error finding container d899153dfd77ff6146c829edc32c72d10639b1bba4b8ceec3626438d5472d141: Status 404 returned error can't find the container with id d899153dfd77ff6146c829edc32c72d10639b1bba4b8ceec3626438d5472d141 Jul 10 18:01:25.416887 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:25.416852 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8f8458798-2klml" event={"ID":"5680deaa-f23f-49c9-8aee-8995859cac4a","Type":"ContainerStarted","Data":"d94300e09420cf04a97b3c36f2fc839c7a2f7680d491c02324ff3364398d4031"} Jul 10 18:01:25.418161 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:25.418133 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerStarted","Data":"d899153dfd77ff6146c829edc32c72d10639b1bba4b8ceec3626438d5472d141"} Jul 10 18:01:25.435616 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:25.435553 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8f8458798-2klml" podStartSLOduration=1.842899199 podStartE2EDuration="3.435535042s" totalImagesPullingTime="1.592635843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:25.433105474 +0000 UTC m=+88.979568640" watchObservedRunningTime="2026-07-10 18:01:25.435535042 +0000 UTC m=+88.981998185" Jul 10 18:01:26.423009 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:26.422979 2569 generic.go:356] "Generic (PLEG): container finished" podID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerID="0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3" exitCode=0 Jul 10 18:01:26.423382 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:26.423071 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerDied","Data":"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3"} Jul 10 18:01:30.443941 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:30.443865 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerStarted","Data":"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1"} Jul 10 18:01:30.444289 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:30.443962 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerStarted","Data":"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e"} Jul 10 18:01:30.978601 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:30.978572 2569 scope.go:117] "RemoveContainer" containerID="60abe0ba19a1a0a16b4efdecec1fe04c61a29f2189d19f7709010607965d094c" Jul 10 18:01:31.450343 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:31.450317 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:01:31.450855 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:31.450385 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-x72w7" event={"ID":"a4f78d82-214c-4612-adc5-32497c685a36","Type":"ContainerStarted","Data":"452a9feba2daf99029df096266bc286ed0b332971eb27b71c85bcb2cd5166cb9"} Jul 10 18:01:31.450855 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:31.450757 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:01:31.467604 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:31.467557 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-x72w7" podStartSLOduration=53.437961339 podStartE2EDuration="58.467539686s" totalImagesPullingTime="5.029578347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:31.466435935 +0000 UTC m=+95.012899078" watchObservedRunningTime="2026-07-10 18:01:31.467539686 +0000 UTC m=+95.014002845" Jul 10 18:01:32.056715 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:32.056686 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-x72w7" Jul 10 18:01:32.457685 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:32.457653 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerStarted","Data":"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6"} Jul 10 18:01:32.458085 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:32.457747 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerStarted","Data":"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d"} Jul 10 18:01:33.465086 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:33.465051 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerStarted","Data":"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19"} Jul 10 18:01:33.465462 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:33.465123 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerStarted","Data":"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea"} Jul 10 18:01:33.495159 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:33.495095 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.539829668 podStartE2EDuration="9.495071452s" totalImagesPullingTime="6.933823198s" totalInitContainerRuntime="21.418586ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:01:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:01:33.492203155 +0000 UTC m=+97.038666332" watchObservedRunningTime="2026-07-10 18:01:33.495071452 +0000 UTC m=+97.041534595" Jul 10 18:01:34.688972 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:34.688937 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:01:35.300875 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:35.300848 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z7dtl" Jul 10 18:01:42.842853 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:42.842800 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:42.843348 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:42.842917 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:01:52.538597 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:52.538503 2569 generic.go:356] "Generic (PLEG): container finished" podID="3404e0d1-d53d-4ccf-8b88-36893095d733" containerID="42ff9c0c70ef0fd1fdb81abaeae17fd1a3f2f9b35caec26c99bbe1b4c3aeba7a" exitCode=0 Jul 10 18:01:52.538597 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:52.538553 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" event={"ID":"3404e0d1-d53d-4ccf-8b88-36893095d733","Type":"ContainerDied","Data":"42ff9c0c70ef0fd1fdb81abaeae17fd1a3f2f9b35caec26c99bbe1b4c3aeba7a"} Jul 10 18:01:52.539120 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:52.538968 2569 scope.go:117] "RemoveContainer" containerID="42ff9c0c70ef0fd1fdb81abaeae17fd1a3f2f9b35caec26c99bbe1b4c3aeba7a" Jul 10 18:01:53.545280 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:53.545244 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-lt7vz" event={"ID":"3404e0d1-d53d-4ccf-8b88-36893095d733","Type":"ContainerStarted","Data":"16b06bd6036c43c1f2ad4d3fae645a59dfa4a8b90e1179769dc119d6b8cae2b9"} Jul 10 18:01:53.683091 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:53.683064 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b875d77bd-5c2fl_14d16fa6-3622-410f-acac-d4ddfe4704b2/router/0.log" Jul 10 18:01:53.695085 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:01:53.695061 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m7jqp_c68aaf65-4c9a-4016-9200-088deed583b9/serve-healthcheck-canary/0.log" Jul 10 18:02:02.848330 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:02.848301 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:02:02.852281 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:02.852256 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8f8458798-2klml" Jul 10 18:02:24.689209 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:24.689160 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:24.705654 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:24.705621 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:25.691561 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:25.691534 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:42.730570 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:42.730528 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 18:02:42.731083 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:42.730952 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19" gracePeriod=600 Jul 10 18:02:42.731083 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:42.730987 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="config-reloader" containerID="cri-o://04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1" gracePeriod=600 Jul 10 18:02:42.731083 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:42.731013 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="kube-rbac-proxy" containerID="cri-o://e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea" gracePeriod=600 Jul 10 18:02:42.731083 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:42.730954 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="kube-rbac-proxy-web" containerID="cri-o://31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6" gracePeriod=600 Jul 10 18:02:42.731289 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:42.730985 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="thanos-sidecar" containerID="cri-o://a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d" gracePeriod=600 Jul 10 18:02:42.731289 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:42.730938 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="prometheus" containerID="cri-o://7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e" gracePeriod=600 Jul 10 18:02:42.983841 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:42.983749 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.065951 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.065919 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-kube-rbac-proxy\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.065951 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.065962 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwdn4\" (UniqueName: \"kubernetes.io/projected/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-kube-api-access-fwdn4\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066227 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066003 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-web-config\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066227 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066037 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-metrics-client-ca\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066227 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066059 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-metrics-client-certs\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066227 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066075 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-thanos-prometheus-http-client-file\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066227 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066108 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066227 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066148 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-config-out\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066227 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066174 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-serving-certs-ca-bundle\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066227 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066218 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-trusted-ca-bundle\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066251 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066280 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-db\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066308 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-config\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066348 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-0\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066395 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-1\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066417 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-tls\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066437 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:02:43.066625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066469 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-tls-assets\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066519 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-2\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066556 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-grpc-tls\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.066625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066582 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-kubelet-serving-ca-bundle\") pod \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\" (UID: \"d0bea28d-d0a5-4898-a05f-bf2aa2628de7\") " Jul 10 18:02:43.067237 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.066914 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-metrics-client-ca\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.068966 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.068933 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:02:43.069753 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.069689 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:02:43.070088 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.069781 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:02:43.070720 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.070663 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:02:43.073609 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.071170 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:02:43.073609 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.072033 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:02:43.073609 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.072536 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:02:43.073609 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.072599 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-kube-api-access-fwdn4" (OuterVolumeSpecName: "kube-api-access-fwdn4") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "kube-api-access-fwdn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:02:43.073609 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.072931 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:02:43.073609 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.073262 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:02:43.074517 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.074487 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:02:43.074629 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.074538 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:02:43.074629 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.074573 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:02:43.074629 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.074603 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:02:43.074629 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.074600 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:02:43.074920 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.074659 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:02:43.075127 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.075103 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-config-out" (OuterVolumeSpecName: "config-out") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:02:43.075960 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.075938 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-config" (OuterVolumeSpecName: "config") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:02:43.084765 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.084732 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-web-config" (OuterVolumeSpecName: "web-config") pod "d0bea28d-d0a5-4898-a05f-bf2aa2628de7" (UID: "d0bea28d-d0a5-4898-a05f-bf2aa2628de7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:02:43.168122 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168078 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168122 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168112 2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-grpc-tls\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168122 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168124 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168137 2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-kube-rbac-proxy\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168147 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwdn4\" (UniqueName: \"kubernetes.io/projected/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-kube-api-access-fwdn4\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168155 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-web-config\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168164 2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-metrics-client-certs\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168172 2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168182 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168190 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-config-out\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168199 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168208 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168220 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168229 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-db\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168238 2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-config\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168246 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168256 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168264 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.168367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.168272 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0bea28d-d0a5-4898-a05f-bf2aa2628de7-tls-assets\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:02:43.756673 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.756643 2569 generic.go:356] "Generic (PLEG): container finished" podID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerID="e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19" exitCode=0 Jul 10 18:02:43.756673 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.756671 2569 generic.go:356] "Generic (PLEG): container finished" podID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerID="e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea" exitCode=0 Jul 10 18:02:43.757121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.756679 2569 generic.go:356] "Generic (PLEG): container finished" podID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerID="31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6" exitCode=0 Jul 10 18:02:43.757121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.756687 2569 generic.go:356] "Generic (PLEG): container finished" podID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerID="a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d" exitCode=0 Jul 10 18:02:43.757121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.756694 2569 generic.go:356] "Generic (PLEG): container finished" podID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerID="04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1" exitCode=0 Jul 10 18:02:43.757121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.756701 2569 generic.go:356] "Generic (PLEG): container finished" podID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerID="7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e" exitCode=0 Jul 10 18:02:43.757121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.756729 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerDied","Data":"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19"} Jul 10 18:02:43.757121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.756752 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.757121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.756833 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerDied","Data":"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea"} Jul 10 18:02:43.757121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.756874 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerDied","Data":"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6"} Jul 10 18:02:43.757121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.756903 2569 scope.go:117] "RemoveContainer" containerID="e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19" Jul 10 18:02:43.757121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.756910 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerDied","Data":"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d"} Jul 10 18:02:43.757121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.757013 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerDied","Data":"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1"} Jul 10 18:02:43.757121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.757049 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerDied","Data":"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e"} Jul 10 18:02:43.757121 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.757090 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d0bea28d-d0a5-4898-a05f-bf2aa2628de7","Type":"ContainerDied","Data":"d899153dfd77ff6146c829edc32c72d10639b1bba4b8ceec3626438d5472d141"} Jul 10 18:02:43.766201 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.766179 2569 scope.go:117] "RemoveContainer" containerID="e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea" Jul 10 18:02:43.773305 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.773286 2569 scope.go:117] "RemoveContainer" containerID="31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6" Jul 10 18:02:43.781588 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.781564 2569 scope.go:117] "RemoveContainer" containerID="a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d" Jul 10 18:02:43.781894 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.781870 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 18:02:43.785979 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.785957 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 18:02:43.789214 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.789195 2569 scope.go:117] "RemoveContainer" containerID="04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1" Jul 10 18:02:43.796326 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.796304 2569 scope.go:117] "RemoveContainer" containerID="7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e" Jul 10 18:02:43.803657 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.803635 2569 scope.go:117] "RemoveContainer" containerID="0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3" Jul 10 18:02:43.809705 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.809682 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 18:02:43.810042 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810029 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="kube-rbac-proxy" Jul 10 18:02:43.810092 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810044 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="kube-rbac-proxy" Jul 10 18:02:43.810092 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810059 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="kube-rbac-proxy-web" Jul 10 18:02:43.810092 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810064 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="kube-rbac-proxy-web" Jul 10 18:02:43.810092 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810074 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="thanos-sidecar" Jul 10 18:02:43.810092 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810079 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="thanos-sidecar" Jul 10 18:02:43.810092 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810089 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="config-reloader" Jul 10 18:02:43.810092 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810094 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="config-reloader" Jul 10 18:02:43.810297 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810102 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="init-config-reloader" Jul 10 18:02:43.810297 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810108 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="init-config-reloader" Jul 10 18:02:43.810297 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810121 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="prometheus" Jul 10 18:02:43.810297 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810127 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="prometheus" Jul 10 18:02:43.810297 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810137 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="kube-rbac-proxy-thanos" Jul 10 18:02:43.810297 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810142 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="kube-rbac-proxy-thanos" Jul 10 18:02:43.810297 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810191 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="config-reloader" Jul 10 18:02:43.810297 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810198 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="kube-rbac-proxy-thanos" Jul 10 18:02:43.810297 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810206 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="kube-rbac-proxy-web" Jul 10 18:02:43.810297 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810212 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="prometheus" Jul 10 18:02:43.810297 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810218 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="thanos-sidecar" Jul 10 18:02:43.810297 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.810223 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" containerName="kube-rbac-proxy" Jul 10 18:02:43.811214 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.811198 2569 scope.go:117] "RemoveContainer" containerID="e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19" Jul 10 18:02:43.811554 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:02:43.811513 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19\": container with ID starting with e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19 not found: ID does not exist" containerID="e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19" Jul 10 18:02:43.811648 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.811563 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19"} err="failed to get container status \"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19\": rpc error: code = NotFound desc = could not find container \"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19\": container with ID starting with e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19 not found: ID does not exist" Jul 10 18:02:43.811648 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.811649 2569 scope.go:117] "RemoveContainer" containerID="e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea" Jul 10 18:02:43.812025 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:02:43.812001 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea\": container with ID starting with e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea not found: ID does not exist" containerID="e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea" Jul 10 18:02:43.812148 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.812031 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea"} err="failed to get container status \"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea\": rpc error: code = NotFound desc = could not find container \"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea\": container with ID starting with e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea not found: ID does not exist" Jul 10 18:02:43.812148 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.812047 2569 scope.go:117] "RemoveContainer" containerID="31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6" Jul 10 18:02:43.812307 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:02:43.812277 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6\": container with ID starting with 31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6 not found: ID does not exist" containerID="31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6" Jul 10 18:02:43.812396 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.812309 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6"} err="failed to get container status \"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6\": rpc error: code = NotFound desc = could not find container \"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6\": container with ID starting with 31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6 not found: ID does not exist" Jul 10 18:02:43.812396 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.812329 2569 scope.go:117] "RemoveContainer" containerID="a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d" Jul 10 18:02:43.812587 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:02:43.812569 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d\": container with ID starting with a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d not found: ID does not exist" containerID="a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d" Jul 10 18:02:43.812637 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.812592 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d"} err="failed to get container status \"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d\": rpc error: code = NotFound desc = could not find container \"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d\": container with ID starting with a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d not found: ID does not exist" Jul 10 18:02:43.812637 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.812610 2569 scope.go:117] "RemoveContainer" containerID="04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1" Jul 10 18:02:43.812886 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:02:43.812868 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1\": container with ID starting with 04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1 not found: ID does not exist" containerID="04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1" Jul 10 18:02:43.812944 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.812891 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1"} err="failed to get container status \"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1\": rpc error: code = NotFound desc = could not find container \"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1\": container with ID starting with 04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1 not found: ID does not exist" Jul 10 18:02:43.812944 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.812905 2569 scope.go:117] "RemoveContainer" containerID="7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e" Jul 10 18:02:43.813161 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:02:43.813144 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e\": container with ID starting with 7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e not found: ID does not exist" containerID="7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e" Jul 10 18:02:43.813230 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.813168 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e"} err="failed to get container status \"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e\": rpc error: code = NotFound desc = could not find container \"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e\": container with ID starting with 7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e not found: ID does not exist" Jul 10 18:02:43.813230 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.813188 2569 scope.go:117] "RemoveContainer" containerID="0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3" Jul 10 18:02:43.813428 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:02:43.813408 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3\": container with ID starting with 0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3 not found: ID does not exist" containerID="0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3" Jul 10 18:02:43.813517 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.813431 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3"} err="failed to get container status \"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3\": rpc error: code = NotFound desc = could not find container \"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3\": container with ID starting with 0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3 not found: ID does not exist" Jul 10 18:02:43.813517 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.813445 2569 scope.go:117] "RemoveContainer" containerID="e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19" Jul 10 18:02:43.813762 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.813736 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19"} err="failed to get container status \"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19\": rpc error: code = NotFound desc = could not find container \"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19\": container with ID starting with e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19 not found: ID does not exist" Jul 10 18:02:43.813762 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.813757 2569 scope.go:117] "RemoveContainer" containerID="e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea" Jul 10 18:02:43.814032 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.814013 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea"} err="failed to get container status \"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea\": rpc error: code = NotFound desc = could not find container \"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea\": container with ID starting with e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea not found: ID does not exist" Jul 10 18:02:43.814097 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.814033 2569 scope.go:117] "RemoveContainer" containerID="31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6" Jul 10 18:02:43.814276 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.814253 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6"} err="failed to get container status \"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6\": rpc error: code = NotFound desc = could not find container \"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6\": container with ID starting with 31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6 not found: ID does not exist" Jul 10 18:02:43.814276 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.814275 2569 scope.go:117] "RemoveContainer" containerID="a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d" Jul 10 18:02:43.814539 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.814518 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d"} err="failed to get container status \"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d\": rpc error: code = NotFound desc = could not find container \"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d\": container with ID starting with a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d not found: ID does not exist" Jul 10 18:02:43.814605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.814540 2569 scope.go:117] "RemoveContainer" containerID="04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1" Jul 10 18:02:43.814844 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.814802 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.814953 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.814874 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1"} err="failed to get container status \"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1\": rpc error: code = NotFound desc = could not find container \"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1\": container with ID starting with 04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1 not found: ID does not exist" Jul 10 18:02:43.814953 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.814902 2569 scope.go:117] "RemoveContainer" containerID="7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e" Jul 10 18:02:43.815592 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.815567 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e"} err="failed to get container status \"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e\": rpc error: code = NotFound desc = could not find container \"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e\": container with ID starting with 7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e not found: ID does not exist" Jul 10 18:02:43.815663 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.815593 2569 scope.go:117] "RemoveContainer" containerID="0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3" Jul 10 18:02:43.815890 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.815870 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3"} err="failed to get container status \"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3\": rpc error: code = NotFound desc = could not find container \"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3\": container with ID starting with 0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3 not found: ID does not exist" Jul 10 18:02:43.815971 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.815891 2569 scope.go:117] "RemoveContainer" containerID="e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19" Jul 10 18:02:43.816163 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.816139 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19"} err="failed to get container status \"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19\": rpc error: code = NotFound desc = could not find container \"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19\": container with ID starting with e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19 not found: ID does not exist" Jul 10 18:02:43.816248 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.816166 2569 scope.go:117] "RemoveContainer" containerID="e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea" Jul 10 18:02:43.816425 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.816390 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea"} err="failed to get container status \"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea\": rpc error: code = NotFound desc = could not find container \"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea\": container with ID starting with e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea not found: ID does not exist" Jul 10 18:02:43.816425 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.816412 2569 scope.go:117] "RemoveContainer" containerID="31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6" Jul 10 18:02:43.816655 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.816632 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6"} err="failed to get container status \"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6\": rpc error: code = NotFound desc = could not find container \"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6\": container with ID starting with 31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6 not found: ID does not exist" Jul 10 18:02:43.816696 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.816658 2569 scope.go:117] "RemoveContainer" containerID="a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d" Jul 10 18:02:43.816928 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.816909 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d"} err="failed to get container status \"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d\": rpc error: code = NotFound desc = could not find container \"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d\": container with ID starting with a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d not found: ID does not exist" Jul 10 18:02:43.817115 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.816931 2569 scope.go:117] "RemoveContainer" containerID="04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1" Jul 10 18:02:43.817202 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.817186 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 18:02:43.817269 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.817197 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1"} err="failed to get container status \"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1\": rpc error: code = NotFound desc = could not find container \"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1\": container with ID starting with 04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1 not found: ID does not exist" Jul 10 18:02:43.817269 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.817214 2569 scope.go:117] "RemoveContainer" containerID="7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e" Jul 10 18:02:43.817269 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.817240 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 18:02:43.817501 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.817390 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 18:02:43.817501 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.817494 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 18:02:43.817710 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.817497 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e"} err="failed to get container status \"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e\": rpc error: code = NotFound desc = could not find container \"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e\": container with ID starting with 7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e not found: ID does not exist" Jul 10 18:02:43.817710 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.817527 2569 scope.go:117] "RemoveContainer" containerID="0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3" Jul 10 18:02:43.817710 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.817639 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 18:02:43.818014 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.817725 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 18:02:43.818014 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.817786 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 18:02:43.818014 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.817826 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3"} err="failed to get container status \"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3\": rpc error: code = NotFound desc = could not find container \"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3\": container with ID starting with 0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3 not found: ID does not exist" Jul 10 18:02:43.818014 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.817849 2569 scope.go:117] "RemoveContainer" containerID="e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19" Jul 10 18:02:43.818265 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.818021 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-m6bp9\"" Jul 10 18:02:43.818265 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.818194 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19"} err="failed to get container status \"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19\": rpc error: code = NotFound desc = could not find container \"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19\": container with ID starting with e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19 not found: ID does not exist" Jul 10 18:02:43.818265 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.818217 2569 scope.go:117] "RemoveContainer" containerID="e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea" Jul 10 18:02:43.818482 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.818445 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea"} err="failed to get container status \"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea\": rpc error: code = NotFound desc = could not find container \"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea\": container with ID starting with e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea not found: ID does not exist" Jul 10 18:02:43.818482 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.818472 2569 scope.go:117] "RemoveContainer" containerID="31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6" Jul 10 18:02:43.818695 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.818630 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 18:02:43.818796 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.818709 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6"} err="failed to get container status \"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6\": rpc error: code = NotFound desc = could not find container \"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6\": container with ID starting with 31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6 not found: ID does not exist" Jul 10 18:02:43.818796 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.818732 2569 scope.go:117] "RemoveContainer" containerID="a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d" Jul 10 18:02:43.818796 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.818746 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 18:02:43.818796 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.818774 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9r9q93jf7jq0b\"" Jul 10 18:02:43.819126 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.818890 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 18:02:43.819126 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.818942 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 18:02:43.819457 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.819433 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d"} err="failed to get container status \"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d\": rpc error: code = NotFound desc = could not find container \"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d\": container with ID starting with a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d not found: ID does not exist" Jul 10 18:02:43.819552 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.819459 2569 scope.go:117] "RemoveContainer" containerID="04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1" Jul 10 18:02:43.819611 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.819571 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 18:02:43.819769 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.819744 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1"} err="failed to get container status \"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1\": rpc error: code = NotFound desc = could not find container \"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1\": container with ID starting with 04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1 not found: ID does not exist" Jul 10 18:02:43.819769 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.819767 2569 scope.go:117] "RemoveContainer" containerID="7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e" Jul 10 18:02:43.820106 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.820083 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e"} err="failed to get container status \"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e\": rpc error: code = NotFound desc = could not find container \"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e\": container with ID starting with 7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e not found: ID does not exist" Jul 10 18:02:43.820106 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.820107 2569 scope.go:117] "RemoveContainer" containerID="0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3" Jul 10 18:02:43.820371 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.820346 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3"} err="failed to get container status \"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3\": rpc error: code = NotFound desc = could not find container \"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3\": container with ID starting with 0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3 not found: ID does not exist" Jul 10 18:02:43.820371 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.820370 2569 scope.go:117] "RemoveContainer" containerID="e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19" Jul 10 18:02:43.820675 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.820649 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19"} err="failed to get container status \"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19\": rpc error: code = NotFound desc = could not find container \"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19\": container with ID starting with e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19 not found: ID does not exist" Jul 10 18:02:43.820764 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.820677 2569 scope.go:117] "RemoveContainer" containerID="e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea" Jul 10 18:02:43.820954 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.820934 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 18:02:43.821057 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.820984 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea"} err="failed to get container status \"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea\": rpc error: code = NotFound desc = could not find container \"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea\": container with ID starting with e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea not found: ID does not exist" Jul 10 18:02:43.821057 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.821013 2569 scope.go:117] "RemoveContainer" containerID="31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6" Jul 10 18:02:43.821275 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.821255 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6"} err="failed to get container status \"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6\": rpc error: code = NotFound desc = could not find container \"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6\": container with ID starting with 31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6 not found: ID does not exist" Jul 10 18:02:43.821343 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.821274 2569 scope.go:117] "RemoveContainer" containerID="a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d" Jul 10 18:02:43.821536 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.821517 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d"} err="failed to get container status \"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d\": rpc error: code = NotFound desc = could not find container \"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d\": container with ID starting with a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d not found: ID does not exist" Jul 10 18:02:43.821606 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.821536 2569 scope.go:117] "RemoveContainer" containerID="04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1" Jul 10 18:02:43.821772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.821754 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1"} err="failed to get container status \"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1\": rpc error: code = NotFound desc = could not find container \"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1\": container with ID starting with 04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1 not found: ID does not exist" Jul 10 18:02:43.821772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.821771 2569 scope.go:117] "RemoveContainer" containerID="7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e" Jul 10 18:02:43.822170 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.822128 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e"} err="failed to get container status \"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e\": rpc error: code = NotFound desc = could not find container \"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e\": container with ID starting with 7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e not found: ID does not exist" Jul 10 18:02:43.822170 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.822155 2569 scope.go:117] "RemoveContainer" containerID="0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3" Jul 10 18:02:43.822417 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.822399 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3"} err="failed to get container status \"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3\": rpc error: code = NotFound desc = could not find container \"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3\": container with ID starting with 0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3 not found: ID does not exist" Jul 10 18:02:43.822474 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.822420 2569 scope.go:117] "RemoveContainer" containerID="e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19" Jul 10 18:02:43.822688 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.822658 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19"} err="failed to get container status \"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19\": rpc error: code = NotFound desc = could not find container \"e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19\": container with ID starting with e25bf9cc52c9d61d935179468c9ccf0553ef0340a2ce3be83558059c1eb19e19 not found: ID does not exist" Jul 10 18:02:43.822786 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.822694 2569 scope.go:117] "RemoveContainer" containerID="e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea" Jul 10 18:02:43.822970 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.822954 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea"} err="failed to get container status \"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea\": rpc error: code = NotFound desc = could not find container \"e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea\": container with ID starting with e2d3314015e2df0802449d6a1806636eb16a3fca3d03cfba10c98121ab84a4ea not found: ID does not exist" Jul 10 18:02:43.823029 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.822972 2569 scope.go:117] "RemoveContainer" containerID="31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6" Jul 10 18:02:43.823254 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.823225 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6"} err="failed to get container status \"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6\": rpc error: code = NotFound desc = could not find container \"31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6\": container with ID starting with 31ac758f7ed9fde4ec4ac02fa0b43d6bc930b804322a5f1f2bee50d8689ef7a6 not found: ID does not exist" Jul 10 18:02:43.823254 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.823252 2569 scope.go:117] "RemoveContainer" containerID="a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d" Jul 10 18:02:43.823519 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.823437 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 18:02:43.823519 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.823502 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d"} err="failed to get container status \"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d\": rpc error: code = NotFound desc = could not find container \"a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d\": container with ID starting with a6d5514c6110d51e0349a97ea698ee9d88cab5d3e77ebc7eaa0c79bedef16b9d not found: ID does not exist" Jul 10 18:02:43.823628 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.823525 2569 scope.go:117] "RemoveContainer" containerID="04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1" Jul 10 18:02:43.823922 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.823902 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1"} err="failed to get container status \"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1\": rpc error: code = NotFound desc = could not find container \"04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1\": container with ID starting with 04818cb1e479eb67700b122009179f8e85d12842565442223f852a7c886061a1 not found: ID does not exist" Jul 10 18:02:43.823998 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.823923 2569 scope.go:117] "RemoveContainer" containerID="7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e" Jul 10 18:02:43.824166 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.824142 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e"} err="failed to get container status \"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e\": rpc error: code = NotFound desc = could not find container \"7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e\": container with ID starting with 7453c162ee7fea9d638f58d767dae20b5afc001c7a25d90ee913017f2bcabb3e not found: ID does not exist" Jul 10 18:02:43.824246 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.824168 2569 scope.go:117] "RemoveContainer" containerID="0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3" Jul 10 18:02:43.824441 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.824419 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3"} err="failed to get container status \"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3\": rpc error: code = NotFound desc = could not find container \"0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3\": container with ID starting with 0874c60069acba1a1064700273f6ea0ed1b5dec8b76a4c403d05d39316827be3 not found: ID does not exist" Jul 10 18:02:43.829559 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.829535 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 18:02:43.976005 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.975962 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976180 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976015 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976180 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976070 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-web-config\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976180 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976095 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976180 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976125 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976180 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976145 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976180 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976161 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976409 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976187 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d23fdb3-3a8e-4274-a6d4-1145264d2167-config-out\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976409 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976225 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976409 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976242 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tn7t\" (UniqueName: \"kubernetes.io/projected/2d23fdb3-3a8e-4274-a6d4-1145264d2167-kube-api-access-8tn7t\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976409 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976267 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976409 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976318 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976409 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976339 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976409 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976361 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976409 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976383 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-config\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976639 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976409 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976639 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976458 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d23fdb3-3a8e-4274-a6d4-1145264d2167-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976639 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976487 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976639 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976507 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:43.976639 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:43.976526 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.077480 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.077446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.077480 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.077486 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-web-config\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.077749 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.077507 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.077749 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.077524 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.077749 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.077542 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.077749 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.077563 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.077977 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.077770 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d23fdb3-3a8e-4274-a6d4-1145264d2167-config-out\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.078153 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.078121 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.078277 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.078166 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tn7t\" (UniqueName: \"kubernetes.io/projected/2d23fdb3-3a8e-4274-a6d4-1145264d2167-kube-api-access-8tn7t\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.078277 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.078208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.078277 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.078260 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.078445 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.078292 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.078445 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.078315 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.078445 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.078316 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.078445 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.078350 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-config\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.078445 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.078386 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.080764 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.080801 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d23fdb3-3a8e-4274-a6d4-1145264d2167-config-out\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.080840 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.081100 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.081107 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.081136 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-web-config\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.081145 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.081200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d23fdb3-3a8e-4274-a6d4-1145264d2167-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.081240 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.081271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.081301 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.081343 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.081597 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.081772 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.081696 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.083251 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.082235 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.083251 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.082318 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.083251 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.082333 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d23fdb3-3a8e-4274-a6d4-1145264d2167-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.083251 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.082389 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.083251 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.082728 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-config\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.083515 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.083430 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.083584 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.083564 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d23fdb3-3a8e-4274-a6d4-1145264d2167-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.083637 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.083608 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.084294 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.084277 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2d23fdb3-3a8e-4274-a6d4-1145264d2167-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.088374 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.088356 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tn7t\" (UniqueName: \"kubernetes.io/projected/2d23fdb3-3a8e-4274-a6d4-1145264d2167-kube-api-access-8tn7t\") pod \"prometheus-k8s-0\" (UID: \"2d23fdb3-3a8e-4274-a6d4-1145264d2167\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.126538 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.126499 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:02:44.278446 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.278416 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 18:02:44.282275 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:02:44.282248 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d23fdb3_3a8e_4274_a6d4_1145264d2167.slice/crio-2bbbf351e2ddc7c29616a2379884e68e10c4f23c5067d3a945584a32825600e6 WatchSource:0}: Error finding container 2bbbf351e2ddc7c29616a2379884e68e10c4f23c5067d3a945584a32825600e6: Status 404 returned error can't find the container with id 2bbbf351e2ddc7c29616a2379884e68e10c4f23c5067d3a945584a32825600e6 Jul 10 18:02:44.763802 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.763699 2569 generic.go:356] "Generic (PLEG): container finished" podID="2d23fdb3-3a8e-4274-a6d4-1145264d2167" containerID="8a3371d56b020540fcad08f4581f8e279f8d1d23c6b2bebaceb16b9ddb46d48c" exitCode=0 Jul 10 18:02:44.763802 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.763770 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d23fdb3-3a8e-4274-a6d4-1145264d2167","Type":"ContainerDied","Data":"8a3371d56b020540fcad08f4581f8e279f8d1d23c6b2bebaceb16b9ddb46d48c"} Jul 10 18:02:44.764280 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.763837 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d23fdb3-3a8e-4274-a6d4-1145264d2167","Type":"ContainerStarted","Data":"2bbbf351e2ddc7c29616a2379884e68e10c4f23c5067d3a945584a32825600e6"} Jul 10 18:02:44.990377 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:44.988774 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0bea28d-d0a5-4898-a05f-bf2aa2628de7" path="/var/lib/kubelet/pods/d0bea28d-d0a5-4898-a05f-bf2aa2628de7/volumes" Jul 10 18:02:45.772277 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:45.772242 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d23fdb3-3a8e-4274-a6d4-1145264d2167","Type":"ContainerStarted","Data":"000bc0cd05595b8f9f18bc4fb5b81e639177e80bb5889a5e48cf7e4072a5b49e"} Jul 10 18:02:45.772711 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:45.772324 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d23fdb3-3a8e-4274-a6d4-1145264d2167","Type":"ContainerStarted","Data":"9d3c0a942697c86098cde8c2cc22c57be36cc8ee7f35d4a5486cd72f7f1490f1"} Jul 10 18:02:45.772711 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:45.772368 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d23fdb3-3a8e-4274-a6d4-1145264d2167","Type":"ContainerStarted","Data":"731c13995a14a577d7d24cd3a81d48f614ec35e08e4f832bb1b826476d31b9a8"} Jul 10 18:02:45.772711 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:45.772408 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d23fdb3-3a8e-4274-a6d4-1145264d2167","Type":"ContainerStarted","Data":"352b543ceb52973ec0307dcfb3d89c507aacf435c19c5a04377d021e5edc43a4"} Jul 10 18:02:45.772711 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:45.772458 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d23fdb3-3a8e-4274-a6d4-1145264d2167","Type":"ContainerStarted","Data":"f0e30b353451e4369d11fdde4d3d261b47e3c106bf608a4cf199c696b2749c71"} Jul 10 18:02:45.772711 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:45.772496 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d23fdb3-3a8e-4274-a6d4-1145264d2167","Type":"ContainerStarted","Data":"4299d38f8b12b093369c44f76503a9ef2597c6597a75bdb6a3ef00239543d613"} Jul 10 18:02:45.808256 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:45.808174 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.787643955 podStartE2EDuration="2.808150784s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.506829ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:02:45.805905505 +0000 UTC m=+169.352368641" watchObservedRunningTime="2026-07-10 18:02:45.808150784 +0000 UTC m=+169.354613925" Jul 10 18:02:49.127061 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:02:49.126964 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:03:44.127240 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:03:44.127201 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:03:44.143125 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:03:44.143099 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:03:45.028253 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:03:45.028225 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 18:04:56.851040 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:04:56.851008 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:04:56.851592 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:04:56.851129 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:04:56.859940 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:04:56.859918 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:04:56.860189 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:04:56.860165 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:04:56.871535 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:04:56.871515 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 18:06:10.818051 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.818022 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq"] Jul 10 18:06:10.821324 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.821308 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.825115 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.825091 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 18:06:10.825240 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.825124 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 18:06:10.825388 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.825368 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 18:06:10.825501 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.825424 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8cfxm\"" Jul 10 18:06:10.825568 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.825518 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 18:06:10.846960 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.846940 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq"] Jul 10 18:06:10.888135 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.888096 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwdxq\" (UniqueName: \"kubernetes.io/projected/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-kube-api-access-dwdxq\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.888135 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.888151 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.888332 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.888212 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-manifests\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.888332 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.888277 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-tmp\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.888468 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.888342 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.989090 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.989057 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwdxq\" (UniqueName: \"kubernetes.io/projected/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-kube-api-access-dwdxq\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.989247 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.989098 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.989247 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.989119 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-manifests\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.989247 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.989157 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-tmp\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.989247 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.989203 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.989644 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.989614 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-manifests\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.991507 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.991482 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-tmp\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.991703 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.991688 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.991853 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.991832 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:10.997260 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:10.997243 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwdxq\" (UniqueName: \"kubernetes.io/projected/d6540ad8-78ea-48aa-8d3c-08413c2b7cc4-kube-api-access-dwdxq\") pod \"opendatahub-operator-controller-manager-bb6dcc87f-l8rcq\" (UID: \"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:11.132169 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:11.132083 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:11.262236 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:11.262206 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq"] Jul 10 18:06:11.265453 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:06:11.265430 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6540ad8_78ea_48aa_8d3c_08413c2b7cc4.slice/crio-88c47346d7167511539e39a0b7a370d36231905397c1a78d9da8d966f6b44e13 WatchSource:0}: Error finding container 88c47346d7167511539e39a0b7a370d36231905397c1a78d9da8d966f6b44e13: Status 404 returned error can't find the container with id 88c47346d7167511539e39a0b7a370d36231905397c1a78d9da8d966f6b44e13 Jul 10 18:06:11.267246 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:11.267227 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:06:11.563434 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:11.563401 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" event={"ID":"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4","Type":"ContainerStarted","Data":"88c47346d7167511539e39a0b7a370d36231905397c1a78d9da8d966f6b44e13"} Jul 10 18:06:15.582660 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:15.582628 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" event={"ID":"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4","Type":"ContainerStarted","Data":"5cdabb16edd383b7109465ca25ee3140a7cb988f9fabf931a4aea2badc0d7d35"} Jul 10 18:06:16.588294 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:16.588261 2569 generic.go:356] "Generic (PLEG): container finished" podID="d6540ad8-78ea-48aa-8d3c-08413c2b7cc4" containerID="5cdabb16edd383b7109465ca25ee3140a7cb988f9fabf931a4aea2badc0d7d35" exitCode=0 Jul 10 18:06:16.588709 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:16.588303 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" event={"ID":"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4","Type":"ContainerDied","Data":"5cdabb16edd383b7109465ca25ee3140a7cb988f9fabf931a4aea2badc0d7d35"} Jul 10 18:06:17.594405 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:17.594368 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" event={"ID":"d6540ad8-78ea-48aa-8d3c-08413c2b7cc4","Type":"ContainerStarted","Data":"51bf9bfe6cef066087c9085d9cf6612b3a8c320e642e2e32e377bd27614a2da9"} Jul 10 18:06:17.594894 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:17.594458 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:17.623026 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:17.622973 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" podStartSLOduration=2.926955495 podStartE2EDuration="7.622959848s" totalImagesPullingTime="3.656395997s" totalInitContainerRuntime="1.039608356s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:06:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:06:17.620428026 +0000 UTC m=+381.166891193" watchObservedRunningTime="2026-07-10 18:06:17.622959848 +0000 UTC m=+381.169422989" Jul 10 18:06:28.600735 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:28.600705 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-bb6dcc87f-l8rcq" Jul 10 18:06:29.831254 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:29.831215 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m"] Jul 10 18:06:29.840371 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:29.840340 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:29.844236 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:29.844213 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 18:06:29.844577 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:29.844555 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 18:06:29.844699 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:29.844624 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:06:29.844960 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:29.844926 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 18:06:29.845094 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:29.845074 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 18:06:29.845162 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:29.845086 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hljzz\"" Jul 10 18:06:29.846779 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:29.846755 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m"] Jul 10 18:06:29.950394 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:29.950351 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqvt9\" (UniqueName: \"kubernetes.io/projected/4d8e150f-f183-4535-8250-d37af204e1c8-kube-api-access-tqvt9\") pod \"lws-controller-manager-6d79ff4f99-xhm9m\" (UID: \"4d8e150f-f183-4535-8250-d37af204e1c8\") " pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:29.950394 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:29.950387 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d8e150f-f183-4535-8250-d37af204e1c8-metrics-cert\") pod \"lws-controller-manager-6d79ff4f99-xhm9m\" (UID: \"4d8e150f-f183-4535-8250-d37af204e1c8\") " pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:29.950596 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:29.950459 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d8e150f-f183-4535-8250-d37af204e1c8-cert\") pod \"lws-controller-manager-6d79ff4f99-xhm9m\" (UID: \"4d8e150f-f183-4535-8250-d37af204e1c8\") " pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:29.950596 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:29.950492 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4d8e150f-f183-4535-8250-d37af204e1c8-manager-config\") pod \"lws-controller-manager-6d79ff4f99-xhm9m\" (UID: \"4d8e150f-f183-4535-8250-d37af204e1c8\") " pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:30.051770 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:30.051735 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvt9\" (UniqueName: \"kubernetes.io/projected/4d8e150f-f183-4535-8250-d37af204e1c8-kube-api-access-tqvt9\") pod \"lws-controller-manager-6d79ff4f99-xhm9m\" (UID: \"4d8e150f-f183-4535-8250-d37af204e1c8\") " pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:30.051770 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:30.051772 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d8e150f-f183-4535-8250-d37af204e1c8-metrics-cert\") pod \"lws-controller-manager-6d79ff4f99-xhm9m\" (UID: \"4d8e150f-f183-4535-8250-d37af204e1c8\") " pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:30.051973 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:30.051804 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d8e150f-f183-4535-8250-d37af204e1c8-cert\") pod \"lws-controller-manager-6d79ff4f99-xhm9m\" (UID: \"4d8e150f-f183-4535-8250-d37af204e1c8\") " pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:30.051973 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:30.051861 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4d8e150f-f183-4535-8250-d37af204e1c8-manager-config\") pod \"lws-controller-manager-6d79ff4f99-xhm9m\" (UID: \"4d8e150f-f183-4535-8250-d37af204e1c8\") " pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:30.052586 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:30.052564 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4d8e150f-f183-4535-8250-d37af204e1c8-manager-config\") pod \"lws-controller-manager-6d79ff4f99-xhm9m\" (UID: \"4d8e150f-f183-4535-8250-d37af204e1c8\") " pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:30.054116 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:30.054088 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d8e150f-f183-4535-8250-d37af204e1c8-cert\") pod \"lws-controller-manager-6d79ff4f99-xhm9m\" (UID: \"4d8e150f-f183-4535-8250-d37af204e1c8\") " pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:30.054215 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:30.054159 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d8e150f-f183-4535-8250-d37af204e1c8-metrics-cert\") pod \"lws-controller-manager-6d79ff4f99-xhm9m\" (UID: \"4d8e150f-f183-4535-8250-d37af204e1c8\") " pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:30.063217 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:30.063193 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqvt9\" (UniqueName: \"kubernetes.io/projected/4d8e150f-f183-4535-8250-d37af204e1c8-kube-api-access-tqvt9\") pod \"lws-controller-manager-6d79ff4f99-xhm9m\" (UID: \"4d8e150f-f183-4535-8250-d37af204e1c8\") " pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:30.151019 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:30.150927 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:30.289468 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:30.289429 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m"] Jul 10 18:06:30.291713 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:06:30.291686 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8e150f_f183_4535_8250_d37af204e1c8.slice/crio-26efc14c714e34c201ef941d762c93116885efb6e06230d61fdbdc169f13413a WatchSource:0}: Error finding container 26efc14c714e34c201ef941d762c93116885efb6e06230d61fdbdc169f13413a: Status 404 returned error can't find the container with id 26efc14c714e34c201ef941d762c93116885efb6e06230d61fdbdc169f13413a Jul 10 18:06:30.653625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:30.653589 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" event={"ID":"4d8e150f-f183-4535-8250-d37af204e1c8","Type":"ContainerStarted","Data":"26efc14c714e34c201ef941d762c93116885efb6e06230d61fdbdc169f13413a"} Jul 10 18:06:31.602033 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.602001 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk"] Jul 10 18:06:31.604399 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.604379 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" Jul 10 18:06:31.607450 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.607329 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f5plj\"" Jul 10 18:06:31.607450 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.607385 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 18:06:31.607450 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.607339 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 18:06:31.616587 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.616565 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk"] Jul 10 18:06:31.766725 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.766688 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de0c53a-b50e-4117-9099-0356c268cde5-tls-certs\") pod \"kube-auth-proxy-5486d894ff-mrqjk\" (UID: \"9de0c53a-b50e-4117-9099-0356c268cde5\") " pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" Jul 10 18:06:31.766725 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.766722 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9de0c53a-b50e-4117-9099-0356c268cde5-tmp\") pod \"kube-auth-proxy-5486d894ff-mrqjk\" (UID: \"9de0c53a-b50e-4117-9099-0356c268cde5\") " pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" Jul 10 18:06:31.766947 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.766753 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxxkn\" (UniqueName: \"kubernetes.io/projected/9de0c53a-b50e-4117-9099-0356c268cde5-kube-api-access-dxxkn\") pod \"kube-auth-proxy-5486d894ff-mrqjk\" (UID: \"9de0c53a-b50e-4117-9099-0356c268cde5\") " pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" Jul 10 18:06:31.867874 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.867765 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de0c53a-b50e-4117-9099-0356c268cde5-tls-certs\") pod \"kube-auth-proxy-5486d894ff-mrqjk\" (UID: \"9de0c53a-b50e-4117-9099-0356c268cde5\") " pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" Jul 10 18:06:31.867874 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.867833 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9de0c53a-b50e-4117-9099-0356c268cde5-tmp\") pod \"kube-auth-proxy-5486d894ff-mrqjk\" (UID: \"9de0c53a-b50e-4117-9099-0356c268cde5\") " pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" Jul 10 18:06:31.867874 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.867872 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxxkn\" (UniqueName: \"kubernetes.io/projected/9de0c53a-b50e-4117-9099-0356c268cde5-kube-api-access-dxxkn\") pod \"kube-auth-proxy-5486d894ff-mrqjk\" (UID: \"9de0c53a-b50e-4117-9099-0356c268cde5\") " pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" Jul 10 18:06:31.870802 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.870772 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9de0c53a-b50e-4117-9099-0356c268cde5-tmp\") pod \"kube-auth-proxy-5486d894ff-mrqjk\" (UID: \"9de0c53a-b50e-4117-9099-0356c268cde5\") " pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" Jul 10 18:06:31.871045 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.871026 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de0c53a-b50e-4117-9099-0356c268cde5-tls-certs\") pod \"kube-auth-proxy-5486d894ff-mrqjk\" (UID: \"9de0c53a-b50e-4117-9099-0356c268cde5\") " pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" Jul 10 18:06:31.875678 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.875656 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxxkn\" (UniqueName: \"kubernetes.io/projected/9de0c53a-b50e-4117-9099-0356c268cde5-kube-api-access-dxxkn\") pod \"kube-auth-proxy-5486d894ff-mrqjk\" (UID: \"9de0c53a-b50e-4117-9099-0356c268cde5\") " pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" Jul 10 18:06:31.918144 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:31.918102 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" Jul 10 18:06:32.802550 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:32.802519 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk"] Jul 10 18:06:32.806349 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:06:32.806321 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de0c53a_b50e_4117_9099_0356c268cde5.slice/crio-b7e736414448be3f7eac48f33a37df65ffee946ff3bc5bd87b9c4ba7c36db1a8 WatchSource:0}: Error finding container b7e736414448be3f7eac48f33a37df65ffee946ff3bc5bd87b9c4ba7c36db1a8: Status 404 returned error can't find the container with id b7e736414448be3f7eac48f33a37df65ffee946ff3bc5bd87b9c4ba7c36db1a8 Jul 10 18:06:33.673081 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:33.673052 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" event={"ID":"9de0c53a-b50e-4117-9099-0356c268cde5","Type":"ContainerStarted","Data":"b7e736414448be3f7eac48f33a37df65ffee946ff3bc5bd87b9c4ba7c36db1a8"} Jul 10 18:06:33.674842 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:33.674788 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" event={"ID":"4d8e150f-f183-4535-8250-d37af204e1c8","Type":"ContainerStarted","Data":"610b7080900bede9995c35e4b1b1e4f17ffe8683b4d2ef601462eb022587016c"} Jul 10 18:06:33.674991 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:33.674973 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:06:33.692662 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:33.692617 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" podStartSLOduration=2.272828068 podStartE2EDuration="4.692602289s" totalImagesPullingTime="2.419774221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:06:33.691060003 +0000 UTC m=+397.237523146" watchObservedRunningTime="2026-07-10 18:06:33.692602289 +0000 UTC m=+397.239065430" Jul 10 18:06:36.690287 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:36.690253 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" event={"ID":"9de0c53a-b50e-4117-9099-0356c268cde5","Type":"ContainerStarted","Data":"0a1ba208169cfc11a8d526c03b98fa07ebe3d1177d55c54cc60c3840663f725e"} Jul 10 18:06:36.706221 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:36.706171 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5486d894ff-mrqjk" podStartSLOduration=2.268328414 podStartE2EDuration="5.706135987s" totalImagesPullingTime="3.437807573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:06:36.705709668 +0000 UTC m=+400.252172865" watchObservedRunningTime="2026-07-10 18:06:36.706135987 +0000 UTC m=+400.252599107" Jul 10 18:06:44.682488 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:06:44.682458 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d79ff4f99-xhm9m" Jul 10 18:08:21.067244 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.067215 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh"] Jul 10 18:08:21.069648 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.069627 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" Jul 10 18:08:21.072327 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.072304 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 18:08:21.072467 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.072375 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 18:08:21.072467 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.072374 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fcbzs\"" Jul 10 18:08:21.073324 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.073302 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 18:08:21.073431 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.073302 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 18:08:21.078604 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.078578 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh"] Jul 10 18:08:21.218159 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.218120 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb142d01-2268-4d06-85a4-a01c17882f4f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qj5mh\" (UID: \"cb142d01-2268-4d06-85a4-a01c17882f4f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" Jul 10 18:08:21.218340 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.218190 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb142d01-2268-4d06-85a4-a01c17882f4f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qj5mh\" (UID: \"cb142d01-2268-4d06-85a4-a01c17882f4f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" Jul 10 18:08:21.218340 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.218227 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxf94\" (UniqueName: \"kubernetes.io/projected/cb142d01-2268-4d06-85a4-a01c17882f4f-kube-api-access-lxf94\") pod \"kuadrant-console-plugin-66b5774985-qj5mh\" (UID: \"cb142d01-2268-4d06-85a4-a01c17882f4f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" Jul 10 18:08:21.319005 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.318916 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb142d01-2268-4d06-85a4-a01c17882f4f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qj5mh\" (UID: \"cb142d01-2268-4d06-85a4-a01c17882f4f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" Jul 10 18:08:21.319005 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.318960 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb142d01-2268-4d06-85a4-a01c17882f4f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qj5mh\" (UID: \"cb142d01-2268-4d06-85a4-a01c17882f4f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" Jul 10 18:08:21.319005 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.318994 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxf94\" (UniqueName: \"kubernetes.io/projected/cb142d01-2268-4d06-85a4-a01c17882f4f-kube-api-access-lxf94\") pod \"kuadrant-console-plugin-66b5774985-qj5mh\" (UID: \"cb142d01-2268-4d06-85a4-a01c17882f4f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" Jul 10 18:08:21.319624 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.319596 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb142d01-2268-4d06-85a4-a01c17882f4f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qj5mh\" (UID: \"cb142d01-2268-4d06-85a4-a01c17882f4f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" Jul 10 18:08:21.321467 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.321444 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb142d01-2268-4d06-85a4-a01c17882f4f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qj5mh\" (UID: \"cb142d01-2268-4d06-85a4-a01c17882f4f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" Jul 10 18:08:21.327798 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.327774 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxf94\" (UniqueName: \"kubernetes.io/projected/cb142d01-2268-4d06-85a4-a01c17882f4f-kube-api-access-lxf94\") pod \"kuadrant-console-plugin-66b5774985-qj5mh\" (UID: \"cb142d01-2268-4d06-85a4-a01c17882f4f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" Jul 10 18:08:21.393012 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.392986 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" Jul 10 18:08:21.518620 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:21.518595 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh"] Jul 10 18:08:21.521140 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:08:21.521113 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb142d01_2268_4d06_85a4_a01c17882f4f.slice/crio-21b33c2fe54b31faa710042391d443f18a406d7976108132a9fa9efde591bf17 WatchSource:0}: Error finding container 21b33c2fe54b31faa710042391d443f18a406d7976108132a9fa9efde591bf17: Status 404 returned error can't find the container with id 21b33c2fe54b31faa710042391d443f18a406d7976108132a9fa9efde591bf17 Jul 10 18:08:22.131229 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:22.131194 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" event={"ID":"cb142d01-2268-4d06-85a4-a01c17882f4f","Type":"ContainerStarted","Data":"21b33c2fe54b31faa710042391d443f18a406d7976108132a9fa9efde591bf17"} Jul 10 18:08:46.251326 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:46.251290 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" event={"ID":"cb142d01-2268-4d06-85a4-a01c17882f4f","Type":"ContainerStarted","Data":"6d6ca4b890020f3188e3c8cc77d6fc3022a69157ad72350fd2e32abed663a7d4"} Jul 10 18:08:46.268030 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:08:46.267974 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qj5mh" podStartSLOduration=1.49761863 podStartE2EDuration="25.267958752s" totalImagesPullingTime="23.770340122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:08:46.266862948 +0000 UTC m=+529.813326082" watchObservedRunningTime="2026-07-10 18:08:46.267958752 +0000 UTC m=+529.814421895" Jul 10 18:09:13.889987 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:13.889953 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:09:14.124533 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:14.124500 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:09:14.124697 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:14.124556 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:09:14.124697 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:14.124618 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tlqnq" Jul 10 18:09:14.127915 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:14.127888 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 18:09:14.186350 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:14.186284 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d8237581-fd48-4441-8145-a1035356193e-config-file\") pod \"limitador-limitador-54d6d988d5-tlqnq\" (UID: \"d8237581-fd48-4441-8145-a1035356193e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tlqnq" Jul 10 18:09:14.186350 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:14.186331 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56zj\" (UniqueName: \"kubernetes.io/projected/d8237581-fd48-4441-8145-a1035356193e-kube-api-access-n56zj\") pod \"limitador-limitador-54d6d988d5-tlqnq\" (UID: \"d8237581-fd48-4441-8145-a1035356193e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tlqnq" Jul 10 18:09:14.287077 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:14.287044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d8237581-fd48-4441-8145-a1035356193e-config-file\") pod \"limitador-limitador-54d6d988d5-tlqnq\" (UID: \"d8237581-fd48-4441-8145-a1035356193e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tlqnq" Jul 10 18:09:14.287077 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:14.287077 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56zj\" (UniqueName: \"kubernetes.io/projected/d8237581-fd48-4441-8145-a1035356193e-kube-api-access-n56zj\") pod \"limitador-limitador-54d6d988d5-tlqnq\" (UID: \"d8237581-fd48-4441-8145-a1035356193e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tlqnq" Jul 10 18:09:14.287656 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:14.287635 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d8237581-fd48-4441-8145-a1035356193e-config-file\") pod \"limitador-limitador-54d6d988d5-tlqnq\" (UID: \"d8237581-fd48-4441-8145-a1035356193e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tlqnq" Jul 10 18:09:14.299299 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:14.299267 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56zj\" (UniqueName: \"kubernetes.io/projected/d8237581-fd48-4441-8145-a1035356193e-kube-api-access-n56zj\") pod \"limitador-limitador-54d6d988d5-tlqnq\" (UID: \"d8237581-fd48-4441-8145-a1035356193e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tlqnq" Jul 10 18:09:14.434944 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:14.434917 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tlqnq" Jul 10 18:09:14.662892 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:14.662865 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:09:14.665127 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:09:14.665103 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8237581_fd48_4441_8145_a1035356193e.slice/crio-541896f3b362fa8bd087c59f23a6cd9110f71016c8511043d4a6cfce8a42e1b1 WatchSource:0}: Error finding container 541896f3b362fa8bd087c59f23a6cd9110f71016c8511043d4a6cfce8a42e1b1: Status 404 returned error can't find the container with id 541896f3b362fa8bd087c59f23a6cd9110f71016c8511043d4a6cfce8a42e1b1 Jul 10 18:09:15.371103 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:15.371068 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tlqnq" event={"ID":"d8237581-fd48-4441-8145-a1035356193e","Type":"ContainerStarted","Data":"541896f3b362fa8bd087c59f23a6cd9110f71016c8511043d4a6cfce8a42e1b1"} Jul 10 18:09:18.385824 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:18.385776 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tlqnq" event={"ID":"d8237581-fd48-4441-8145-a1035356193e","Type":"ContainerStarted","Data":"a2117b69c684626812efc0544c716bb1782fa45918672e62312c673803ab23c1"} Jul 10 18:09:18.386218 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:18.385879 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tlqnq" Jul 10 18:09:18.407796 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:18.407735 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-tlqnq" podStartSLOduration=2.353730063 podStartE2EDuration="5.407713813s" totalImagesPullingTime="3.05398375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:09:18.405975593 +0000 UTC m=+561.952438755" watchObservedRunningTime="2026-07-10 18:09:18.407713813 +0000 UTC m=+561.954176957" Jul 10 18:09:29.390741 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:29.390712 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tlqnq" Jul 10 18:09:47.421187 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:47.421151 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66f4f94d94-kj7hn"] Jul 10 18:09:47.424509 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:47.424490 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66f4f94d94-kj7hn" Jul 10 18:09:47.428212 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:47.428187 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hp22f\"" Jul 10 18:09:47.441680 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:47.441660 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66f4f94d94-kj7hn"] Jul 10 18:09:47.588609 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:47.588575 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z582m\" (UniqueName: \"kubernetes.io/projected/0f0f5acf-a6a1-4e11-9bff-054c02d874b8-kube-api-access-z582m\") pod \"authorino-66f4f94d94-kj7hn\" (UID: \"0f0f5acf-a6a1-4e11-9bff-054c02d874b8\") " pod="kuadrant-system/authorino-66f4f94d94-kj7hn" Jul 10 18:09:47.689897 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:47.689800 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z582m\" (UniqueName: \"kubernetes.io/projected/0f0f5acf-a6a1-4e11-9bff-054c02d874b8-kube-api-access-z582m\") pod \"authorino-66f4f94d94-kj7hn\" (UID: \"0f0f5acf-a6a1-4e11-9bff-054c02d874b8\") " pod="kuadrant-system/authorino-66f4f94d94-kj7hn" Jul 10 18:09:47.710483 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:47.710455 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z582m\" (UniqueName: \"kubernetes.io/projected/0f0f5acf-a6a1-4e11-9bff-054c02d874b8-kube-api-access-z582m\") pod \"authorino-66f4f94d94-kj7hn\" (UID: \"0f0f5acf-a6a1-4e11-9bff-054c02d874b8\") " pod="kuadrant-system/authorino-66f4f94d94-kj7hn" Jul 10 18:09:47.733147 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:47.733121 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66f4f94d94-kj7hn" Jul 10 18:09:47.793492 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:47.793460 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66f4f94d94-kj7hn"] Jul 10 18:09:47.870063 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:47.870028 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66f4f94d94-kj7hn"] Jul 10 18:09:47.875215 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:09:47.875185 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f0f5acf_a6a1_4e11_9bff_054c02d874b8.slice/crio-5d506f44578a0895a5eff4f6e0613c3c65a25507702de793da4fdf66523db7f0 WatchSource:0}: Error finding container 5d506f44578a0895a5eff4f6e0613c3c65a25507702de793da4fdf66523db7f0: Status 404 returned error can't find the container with id 5d506f44578a0895a5eff4f6e0613c3c65a25507702de793da4fdf66523db7f0 Jul 10 18:09:48.516218 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:48.516175 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66f4f94d94-kj7hn" event={"ID":"0f0f5acf-a6a1-4e11-9bff-054c02d874b8","Type":"ContainerStarted","Data":"5d506f44578a0895a5eff4f6e0613c3c65a25507702de793da4fdf66523db7f0"} Jul 10 18:09:51.533637 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:51.533600 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66f4f94d94-kj7hn" event={"ID":"0f0f5acf-a6a1-4e11-9bff-054c02d874b8","Type":"ContainerStarted","Data":"d8430aa139ce5b685c58136c7d7c746800c51143d4cb2f6279d69356eb1ca6b2"} Jul 10 18:09:51.534045 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:51.533682 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-66f4f94d94-kj7hn" podUID="0f0f5acf-a6a1-4e11-9bff-054c02d874b8" containerName="authorino" containerID="cri-o://d8430aa139ce5b685c58136c7d7c746800c51143d4cb2f6279d69356eb1ca6b2" gracePeriod=30 Jul 10 18:09:51.572031 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:51.571991 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66f4f94d94-kj7hn" podStartSLOduration=1.078886645 podStartE2EDuration="4.571976857s" totalImagesPullingTime="3.493090212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:09:51.571845732 +0000 UTC m=+595.118308864" watchObservedRunningTime="2026-07-10 18:09:51.571976857 +0000 UTC m=+595.118439998" Jul 10 18:09:51.771551 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:51.771527 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66f4f94d94-kj7hn" Jul 10 18:09:51.831075 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:51.831044 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z582m\" (UniqueName: \"kubernetes.io/projected/0f0f5acf-a6a1-4e11-9bff-054c02d874b8-kube-api-access-z582m\") pod \"0f0f5acf-a6a1-4e11-9bff-054c02d874b8\" (UID: \"0f0f5acf-a6a1-4e11-9bff-054c02d874b8\") " Jul 10 18:09:51.833180 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:51.833149 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0f5acf-a6a1-4e11-9bff-054c02d874b8-kube-api-access-z582m" (OuterVolumeSpecName: "kube-api-access-z582m") pod "0f0f5acf-a6a1-4e11-9bff-054c02d874b8" (UID: "0f0f5acf-a6a1-4e11-9bff-054c02d874b8"). InnerVolumeSpecName "kube-api-access-z582m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:09:51.931637 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:51.931572 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z582m\" (UniqueName: \"kubernetes.io/projected/0f0f5acf-a6a1-4e11-9bff-054c02d874b8-kube-api-access-z582m\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:09:52.539416 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:52.539381 2569 generic.go:356] "Generic (PLEG): container finished" podID="0f0f5acf-a6a1-4e11-9bff-054c02d874b8" containerID="d8430aa139ce5b685c58136c7d7c746800c51143d4cb2f6279d69356eb1ca6b2" exitCode=0 Jul 10 18:09:52.539859 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:52.539421 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66f4f94d94-kj7hn" Jul 10 18:09:52.539859 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:52.539466 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66f4f94d94-kj7hn" event={"ID":"0f0f5acf-a6a1-4e11-9bff-054c02d874b8","Type":"ContainerDied","Data":"d8430aa139ce5b685c58136c7d7c746800c51143d4cb2f6279d69356eb1ca6b2"} Jul 10 18:09:52.539859 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:52.539504 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66f4f94d94-kj7hn" event={"ID":"0f0f5acf-a6a1-4e11-9bff-054c02d874b8","Type":"ContainerDied","Data":"5d506f44578a0895a5eff4f6e0613c3c65a25507702de793da4fdf66523db7f0"} Jul 10 18:09:52.539859 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:52.539521 2569 scope.go:117] "RemoveContainer" containerID="d8430aa139ce5b685c58136c7d7c746800c51143d4cb2f6279d69356eb1ca6b2" Jul 10 18:09:52.548274 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:52.548259 2569 scope.go:117] "RemoveContainer" containerID="d8430aa139ce5b685c58136c7d7c746800c51143d4cb2f6279d69356eb1ca6b2" Jul 10 18:09:52.548511 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:09:52.548493 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8430aa139ce5b685c58136c7d7c746800c51143d4cb2f6279d69356eb1ca6b2\": container with ID starting with d8430aa139ce5b685c58136c7d7c746800c51143d4cb2f6279d69356eb1ca6b2 not found: ID does not exist" containerID="d8430aa139ce5b685c58136c7d7c746800c51143d4cb2f6279d69356eb1ca6b2" Jul 10 18:09:52.548566 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:52.548517 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8430aa139ce5b685c58136c7d7c746800c51143d4cb2f6279d69356eb1ca6b2"} err="failed to get container status \"d8430aa139ce5b685c58136c7d7c746800c51143d4cb2f6279d69356eb1ca6b2\": rpc error: code = NotFound desc = could not find container \"d8430aa139ce5b685c58136c7d7c746800c51143d4cb2f6279d69356eb1ca6b2\": container with ID starting with d8430aa139ce5b685c58136c7d7c746800c51143d4cb2f6279d69356eb1ca6b2 not found: ID does not exist" Jul 10 18:09:52.559028 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:52.559006 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66f4f94d94-kj7hn"] Jul 10 18:09:52.562667 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:52.562647 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-66f4f94d94-kj7hn"] Jul 10 18:09:52.984622 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:52.984592 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f0f5acf-a6a1-4e11-9bff-054c02d874b8" path="/var/lib/kubelet/pods/0f0f5acf-a6a1-4e11-9bff-054c02d874b8/volumes" Jul 10 18:09:56.887995 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:56.887966 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:09:56.891081 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:56.891056 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:09:56.893769 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:56.893746 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:09:56.897217 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:09:56.897197 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:10:30.162976 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:10:30.162941 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:10:47.262830 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:10:47.262766 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:10:56.439803 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:10:56.439766 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:11:01.742588 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.742560 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw"] Jul 10 18:11:01.742980 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.742900 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f0f5acf-a6a1-4e11-9bff-054c02d874b8" containerName="authorino" Jul 10 18:11:01.742980 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.742911 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f0f5acf-a6a1-4e11-9bff-054c02d874b8" containerName="authorino" Jul 10 18:11:01.742980 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.742976 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f0f5acf-a6a1-4e11-9bff-054c02d874b8" containerName="authorino" Jul 10 18:11:01.744800 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.744784 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.748893 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.748871 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n65md\"" Jul 10 18:11:01.749028 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.748874 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 18:11:01.749028 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.748874 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 10 18:11:01.749028 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.748880 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 18:11:01.758804 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.758784 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw"] Jul 10 18:11:01.851375 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.851345 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27ce69f0-5289-4897-99c2-5f336c41b810-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.851375 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.851377 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27ce69f0-5289-4897-99c2-5f336c41b810-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.851583 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.851405 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27ce69f0-5289-4897-99c2-5f336c41b810-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.851583 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.851455 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27ce69f0-5289-4897-99c2-5f336c41b810-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.851583 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.851493 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2tcl\" (UniqueName: \"kubernetes.io/projected/27ce69f0-5289-4897-99c2-5f336c41b810-kube-api-access-n2tcl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.851583 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.851534 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27ce69f0-5289-4897-99c2-5f336c41b810-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.952774 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.952739 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27ce69f0-5289-4897-99c2-5f336c41b810-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.952940 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.952783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2tcl\" (UniqueName: \"kubernetes.io/projected/27ce69f0-5289-4897-99c2-5f336c41b810-kube-api-access-n2tcl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.952940 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.952852 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27ce69f0-5289-4897-99c2-5f336c41b810-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.952940 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.952905 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27ce69f0-5289-4897-99c2-5f336c41b810-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.952940 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.952925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27ce69f0-5289-4897-99c2-5f336c41b810-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.953137 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.952945 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27ce69f0-5289-4897-99c2-5f336c41b810-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.953327 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.953303 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27ce69f0-5289-4897-99c2-5f336c41b810-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.953396 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.953354 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27ce69f0-5289-4897-99c2-5f336c41b810-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.953396 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.953380 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27ce69f0-5289-4897-99c2-5f336c41b810-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.955039 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.955016 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27ce69f0-5289-4897-99c2-5f336c41b810-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.955201 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.955183 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27ce69f0-5289-4897-99c2-5f336c41b810-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:01.961298 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:01.961266 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2tcl\" (UniqueName: \"kubernetes.io/projected/27ce69f0-5289-4897-99c2-5f336c41b810-kube-api-access-n2tcl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4fpnw\" (UID: \"27ce69f0-5289-4897-99c2-5f336c41b810\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:02.054781 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:02.054752 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:02.192559 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:02.192525 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw"] Jul 10 18:11:02.195083 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:11:02.195054 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ce69f0_5289_4897_99c2_5f336c41b810.slice/crio-0439e7ce0e9733af74cadde2bf403cfc746e99228d0a28e1f5e78df7d3622f64 WatchSource:0}: Error finding container 0439e7ce0e9733af74cadde2bf403cfc746e99228d0a28e1f5e78df7d3622f64: Status 404 returned error can't find the container with id 0439e7ce0e9733af74cadde2bf403cfc746e99228d0a28e1f5e78df7d3622f64 Jul 10 18:11:02.858554 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:02.858523 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" event={"ID":"27ce69f0-5289-4897-99c2-5f336c41b810","Type":"ContainerStarted","Data":"0439e7ce0e9733af74cadde2bf403cfc746e99228d0a28e1f5e78df7d3622f64"} Jul 10 18:11:02.969501 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:02.969462 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:11:03.864472 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:03.864440 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" event={"ID":"27ce69f0-5289-4897-99c2-5f336c41b810","Type":"ContainerStarted","Data":"738fcf29845482a44b1ab6987c0ae7fd3890da57d03d161494093de0188b4fed"} Jul 10 18:11:03.885019 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:03.884975 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" podStartSLOduration=1.863532598 podStartE2EDuration="2.884958336s" totalImagesPullingTime="1.021425738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:11:03.883486133 +0000 UTC m=+667.429949274" watchObservedRunningTime="2026-07-10 18:11:03.884958336 +0000 UTC m=+667.431421477" Jul 10 18:11:10.028149 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:10.028115 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:11:12.055799 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:12.055767 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:12.056189 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:12.055846 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:12.068562 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:12.068535 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:12.913884 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:12.913858 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4fpnw" Jul 10 18:11:29.820787 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:29.820754 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:11:40.337007 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.336975 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8"] Jul 10 18:11:40.340309 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.340295 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.343270 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.343252 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 10 18:11:40.353495 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.353469 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8"] Jul 10 18:11:40.500154 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.500122 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4438805-1b27-49c7-a434-b6ae614fb73d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.500345 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.500169 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4438805-1b27-49c7-a434-b6ae614fb73d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.500345 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.500189 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4438805-1b27-49c7-a434-b6ae614fb73d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.500345 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.500283 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs88v\" (UniqueName: \"kubernetes.io/projected/a4438805-1b27-49c7-a434-b6ae614fb73d-kube-api-access-gs88v\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.500508 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.500364 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4438805-1b27-49c7-a434-b6ae614fb73d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.500508 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.500398 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4438805-1b27-49c7-a434-b6ae614fb73d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.601138 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.601062 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4438805-1b27-49c7-a434-b6ae614fb73d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.601138 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.601109 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4438805-1b27-49c7-a434-b6ae614fb73d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.601138 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.601125 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4438805-1b27-49c7-a434-b6ae614fb73d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.601330 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.601166 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs88v\" (UniqueName: \"kubernetes.io/projected/a4438805-1b27-49c7-a434-b6ae614fb73d-kube-api-access-gs88v\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.601330 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.601220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4438805-1b27-49c7-a434-b6ae614fb73d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.601330 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.601246 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4438805-1b27-49c7-a434-b6ae614fb73d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.601597 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.601565 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4438805-1b27-49c7-a434-b6ae614fb73d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.601723 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.601598 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4438805-1b27-49c7-a434-b6ae614fb73d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.601723 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.601641 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4438805-1b27-49c7-a434-b6ae614fb73d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.603259 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.603233 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4438805-1b27-49c7-a434-b6ae614fb73d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.603495 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.603477 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4438805-1b27-49c7-a434-b6ae614fb73d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.608860 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.608842 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs88v\" (UniqueName: \"kubernetes.io/projected/a4438805-1b27-49c7-a434-b6ae614fb73d-kube-api-access-gs88v\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8\" (UID: \"a4438805-1b27-49c7-a434-b6ae614fb73d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.649469 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.649444 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:40.779299 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.779275 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8"] Jul 10 18:11:40.781182 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:11:40.781158 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4438805_1b27_49c7_a434_b6ae614fb73d.slice/crio-4712abc67107d5db008ab04b564cd51d4c3af82d0380e1402aef29b9a5e1f586 WatchSource:0}: Error finding container 4712abc67107d5db008ab04b564cd51d4c3af82d0380e1402aef29b9a5e1f586: Status 404 returned error can't find the container with id 4712abc67107d5db008ab04b564cd51d4c3af82d0380e1402aef29b9a5e1f586 Jul 10 18:11:40.782486 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:40.782470 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:11:41.031532 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:41.031508 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" event={"ID":"a4438805-1b27-49c7-a434-b6ae614fb73d","Type":"ContainerStarted","Data":"4712abc67107d5db008ab04b564cd51d4c3af82d0380e1402aef29b9a5e1f586"} Jul 10 18:11:42.042868 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:42.042834 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" event={"ID":"a4438805-1b27-49c7-a434-b6ae614fb73d","Type":"ContainerStarted","Data":"8a2214416fb96e9bddf536b820d5816adfed5973a6707a3d6e2ad660238f60fd"} Jul 10 18:11:42.076605 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:42.076560 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" podStartSLOduration=1.848326307 podStartE2EDuration="2.076546092s" totalImagesPullingTime="228.219785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:11:42.0743469 +0000 UTC m=+705.620810043" watchObservedRunningTime="2026-07-10 18:11:42.076546092 +0000 UTC m=+705.623009234" Jul 10 18:11:43.076119 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:43.076089 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:11:50.650014 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:50.649929 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:50.650014 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:50.649973 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:50.662394 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:50.662370 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:11:51.095110 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:11:51.095088 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8" Jul 10 18:13:21.454660 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:13:21.454585 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:13:31.863843 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:13:31.863796 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:13:42.251137 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:13:42.251101 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:13:50.947897 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:13:50.947866 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:14:01.540071 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:14:01.540032 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:14:28.356885 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:14:28.356850 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:14:41.362558 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:14:41.362525 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:14:56.917144 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:14:56.917061 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:14:56.922600 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:14:56.922576 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:14:56.923292 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:14:56.923275 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:14:56.928685 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:14:56.928669 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:15:16.045984 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:15:16.045950 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:15:25.744328 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:15:25.744294 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:15:36.439829 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:15:36.439783 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:15:44.571870 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:15:44.571836 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:15:56.749625 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:15:56.749585 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:16:07.149086 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:16:07.149054 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:16:18.647681 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:16:18.647606 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:16:28.841709 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:16:28.841676 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:16:50.548096 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:16:50.548061 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:17:02.144848 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:17:02.144804 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:17:25.251330 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:17:25.251297 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:17:38.234241 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:17:38.234204 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:17:53.743529 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:17:53.743449 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:18:04.041034 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:18:04.040997 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:18:14.135525 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:18:14.135488 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:18:24.137653 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:18:24.137617 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:18:34.043336 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:18:34.043302 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:19:56.946347 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:19:56.946002 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:19:56.953258 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:19:56.953234 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:19:56.953904 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:19:56.953885 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:19:56.959256 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:19:56.959240 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:20:19.849092 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:20:19.849057 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:20:32.256497 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:20:32.256459 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:20:42.041855 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:20:42.041821 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:20:53.443883 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:20:53.443851 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:21:03.642003 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:21:03.641972 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:21:15.240842 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:21:15.240797 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:21:24.253788 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:21:24.253750 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:21:34.737799 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:21:34.737763 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:21:46.233746 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:21:46.233709 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:21:54.352677 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:21:54.352644 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:22:05.845575 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:22:05.845541 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:22:32.133200 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:22:32.133122 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:22:39.755452 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:22:39.755416 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:22:52.940395 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:22:52.940360 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:23:03.342660 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:23:03.342627 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:23:20.950245 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:23:20.950213 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:23:34.661081 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:23:34.661050 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:23:44.855382 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:23:44.855349 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:23:56.261750 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:23:56.261667 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:24:09.561800 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:24:09.561769 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:24:28.446276 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:24:28.446234 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:24:46.442540 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:24:46.442502 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:24:54.839689 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:24:54.839657 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:24:56.977410 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:24:56.977384 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:24:56.982684 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:24:56.982660 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:24:56.983972 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:24:56.983950 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:24:56.988404 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:24:56.988387 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:25:06.741395 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:25:06.741361 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:25:23.047637 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:25:23.047564 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:25:36.540246 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:25:36.540210 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:25:48.556523 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:25:48.556489 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:26:03.265959 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:26:03.265927 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:26:13.960065 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:26:13.960019 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:26:30.363216 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:26:30.363183 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:26:57.038204 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:26:57.038164 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:27:08.549716 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:27:08.549685 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:28:28.044923 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:28:28.044839 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:28:37.744069 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:28:37.744038 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:28:50.137931 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:28:50.137900 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:28:59.443603 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:28:59.443570 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:29:31.748195 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:31.748161 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:29:40.142359 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.142322 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8"] Jul 10 18:29:40.145798 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.145777 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.148332 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.148309 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-95637065\"/\"openshift-service-ca.crt\"" Jul 10 18:29:40.149449 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.149427 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-95637065\"/\"default-dockercfg-hxljx\"" Jul 10 18:29:40.149558 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.149507 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-95637065\"/\"kube-root-ca.crt\"" Jul 10 18:29:40.149623 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.149569 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-95637065\"/\"e2e-lifecycle-model-95637065-kserve-self-signed-certs\"" Jul 10 18:29:40.154295 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.154276 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8"] Jul 10 18:29:40.318827 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.318775 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-home\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.318985 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.318845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck4tb\" (UniqueName: \"kubernetes.io/projected/c24c01a3-5f5c-4000-afc3-ace373bf337b-kube-api-access-ck4tb\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.318985 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.318873 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-tmp-dir\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.318985 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.318912 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-dshm\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.318985 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.318953 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c24c01a3-5f5c-4000-afc3-ace373bf337b-tls-certs\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.318985 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.318984 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-model-cache\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.420293 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.420218 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck4tb\" (UniqueName: \"kubernetes.io/projected/c24c01a3-5f5c-4000-afc3-ace373bf337b-kube-api-access-ck4tb\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.420293 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.420257 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-tmp-dir\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.420517 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.420308 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-dshm\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.420517 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.420325 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c24c01a3-5f5c-4000-afc3-ace373bf337b-tls-certs\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.420517 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.420360 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-model-cache\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.420517 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.420391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-home\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.420851 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.420797 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-tmp-dir\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.420851 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.420842 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-model-cache\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.420964 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.420903 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-home\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.422630 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.422604 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-dshm\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.422740 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.422717 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c24c01a3-5f5c-4000-afc3-ace373bf337b-tls-certs\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.427975 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.427955 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck4tb\" (UniqueName: \"kubernetes.io/projected/c24c01a3-5f5c-4000-afc3-ace373bf337b-kube-api-access-ck4tb\") pod \"e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.455110 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.455082 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:40.581006 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.580916 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8"] Jul 10 18:29:40.583930 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:29:40.583898 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc24c01a3_5f5c_4000_afc3_ace373bf337b.slice/crio-f3466853f7708258f6c843b11cad81791d297643e7b324942fb045816bbe87fe WatchSource:0}: Error finding container f3466853f7708258f6c843b11cad81791d297643e7b324942fb045816bbe87fe: Status 404 returned error can't find the container with id f3466853f7708258f6c843b11cad81791d297643e7b324942fb045816bbe87fe Jul 10 18:29:40.745594 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.745517 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" event={"ID":"c24c01a3-5f5c-4000-afc3-ace373bf337b","Type":"ContainerStarted","Data":"e4afd7f3fd46c2312938e365e7c03e1c6a7c2b6854fb4dee6c2ab953c731cd56"} Jul 10 18:29:40.745594 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.745553 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" event={"ID":"c24c01a3-5f5c-4000-afc3-ace373bf337b","Type":"ContainerStarted","Data":"f3466853f7708258f6c843b11cad81791d297643e7b324942fb045816bbe87fe"} Jul 10 18:29:40.761886 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:40.761840 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" podStartSLOduration=0.761797851 podStartE2EDuration="761.797851ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:40.760790256 +0000 UTC m=+1784.307253397" watchObservedRunningTime="2026-07-10 18:29:40.761797851 +0000 UTC m=+1784.308260992" Jul 10 18:29:44.835197 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:44.835163 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:29:50.455712 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:50.455662 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:50.455712 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:50.455723 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:50.467977 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:50.467951 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:50.799891 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:50.799779 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:29:57.004750 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:57.004720 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:29:57.009991 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:57.009968 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:29:57.010090 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:57.009968 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:29:57.014931 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:57.014915 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:29:58.244482 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:29:58.244441 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:30:11.640702 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:11.640666 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8"] Jul 10 18:30:12.740293 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:12.740261 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:30:26.660460 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:26.660396 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" podUID="c24c01a3-5f5c-4000-afc3-ace373bf337b" containerName="main" containerID="cri-o://e4afd7f3fd46c2312938e365e7c03e1c6a7c2b6854fb4dee6c2ab953c731cd56" gracePeriod=45 Jul 10 18:30:26.903195 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:26.903171 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:30:26.961443 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:26.961363 2569 generic.go:356] "Generic (PLEG): container finished" podID="c24c01a3-5f5c-4000-afc3-ace373bf337b" containerID="e4afd7f3fd46c2312938e365e7c03e1c6a7c2b6854fb4dee6c2ab953c731cd56" exitCode=0 Jul 10 18:30:26.961443 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:26.961407 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" Jul 10 18:30:26.961617 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:26.961446 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" event={"ID":"c24c01a3-5f5c-4000-afc3-ace373bf337b","Type":"ContainerDied","Data":"e4afd7f3fd46c2312938e365e7c03e1c6a7c2b6854fb4dee6c2ab953c731cd56"} Jul 10 18:30:26.961617 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:26.961498 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8" event={"ID":"c24c01a3-5f5c-4000-afc3-ace373bf337b","Type":"ContainerDied","Data":"f3466853f7708258f6c843b11cad81791d297643e7b324942fb045816bbe87fe"} Jul 10 18:30:26.961617 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:26.961521 2569 scope.go:117] "RemoveContainer" containerID="e4afd7f3fd46c2312938e365e7c03e1c6a7c2b6854fb4dee6c2ab953c731cd56" Jul 10 18:30:26.970201 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:26.970185 2569 scope.go:117] "RemoveContainer" containerID="e4afd7f3fd46c2312938e365e7c03e1c6a7c2b6854fb4dee6c2ab953c731cd56" Jul 10 18:30:26.970482 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:30:26.970454 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4afd7f3fd46c2312938e365e7c03e1c6a7c2b6854fb4dee6c2ab953c731cd56\": container with ID starting with e4afd7f3fd46c2312938e365e7c03e1c6a7c2b6854fb4dee6c2ab953c731cd56 not found: ID does not exist" containerID="e4afd7f3fd46c2312938e365e7c03e1c6a7c2b6854fb4dee6c2ab953c731cd56" Jul 10 18:30:26.970551 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:26.970484 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4afd7f3fd46c2312938e365e7c03e1c6a7c2b6854fb4dee6c2ab953c731cd56"} err="failed to get container status \"e4afd7f3fd46c2312938e365e7c03e1c6a7c2b6854fb4dee6c2ab953c731cd56\": rpc error: code = NotFound desc = could not find container \"e4afd7f3fd46c2312938e365e7c03e1c6a7c2b6854fb4dee6c2ab953c731cd56\": container with ID starting with e4afd7f3fd46c2312938e365e7c03e1c6a7c2b6854fb4dee6c2ab953c731cd56 not found: ID does not exist" Jul 10 18:30:27.047060 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.047031 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-dshm\") pod \"c24c01a3-5f5c-4000-afc3-ace373bf337b\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " Jul 10 18:30:27.047060 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.047065 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-model-cache\") pod \"c24c01a3-5f5c-4000-afc3-ace373bf337b\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " Jul 10 18:30:27.047283 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.047104 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck4tb\" (UniqueName: \"kubernetes.io/projected/c24c01a3-5f5c-4000-afc3-ace373bf337b-kube-api-access-ck4tb\") pod \"c24c01a3-5f5c-4000-afc3-ace373bf337b\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " Jul 10 18:30:27.047283 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.047121 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c24c01a3-5f5c-4000-afc3-ace373bf337b-tls-certs\") pod \"c24c01a3-5f5c-4000-afc3-ace373bf337b\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " Jul 10 18:30:27.047283 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.047168 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-tmp-dir\") pod \"c24c01a3-5f5c-4000-afc3-ace373bf337b\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " Jul 10 18:30:27.047283 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.047201 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-home\") pod \"c24c01a3-5f5c-4000-afc3-ace373bf337b\" (UID: \"c24c01a3-5f5c-4000-afc3-ace373bf337b\") " Jul 10 18:30:27.047479 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.047402 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-model-cache" (OuterVolumeSpecName: "model-cache") pod "c24c01a3-5f5c-4000-afc3-ace373bf337b" (UID: "c24c01a3-5f5c-4000-afc3-ace373bf337b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:30:27.047715 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.047671 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c24c01a3-5f5c-4000-afc3-ace373bf337b" (UID: "c24c01a3-5f5c-4000-afc3-ace373bf337b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:30:27.049672 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.047907 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-home" (OuterVolumeSpecName: "home") pod "c24c01a3-5f5c-4000-afc3-ace373bf337b" (UID: "c24c01a3-5f5c-4000-afc3-ace373bf337b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:30:27.049672 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.048042 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-model-cache\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:30:27.049905 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.049871 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c24c01a3-5f5c-4000-afc3-ace373bf337b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c24c01a3-5f5c-4000-afc3-ace373bf337b" (UID: "c24c01a3-5f5c-4000-afc3-ace373bf337b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:30:27.050637 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.050613 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24c01a3-5f5c-4000-afc3-ace373bf337b-kube-api-access-ck4tb" (OuterVolumeSpecName: "kube-api-access-ck4tb") pod "c24c01a3-5f5c-4000-afc3-ace373bf337b" (UID: "c24c01a3-5f5c-4000-afc3-ace373bf337b"). InnerVolumeSpecName "kube-api-access-ck4tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:30:27.058038 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.058012 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-dshm" (OuterVolumeSpecName: "dshm") pod "c24c01a3-5f5c-4000-afc3-ace373bf337b" (UID: "c24c01a3-5f5c-4000-afc3-ace373bf337b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:30:27.149271 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.149237 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-dshm\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:30:27.149271 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.149265 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck4tb\" (UniqueName: \"kubernetes.io/projected/c24c01a3-5f5c-4000-afc3-ace373bf337b-kube-api-access-ck4tb\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:30:27.149446 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.149279 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c24c01a3-5f5c-4000-afc3-ace373bf337b-tls-certs\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:30:27.149446 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.149292 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-tmp-dir\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:30:27.149446 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.149305 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c24c01a3-5f5c-4000-afc3-ace373bf337b-home\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:30:27.283873 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.283843 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8"] Jul 10 18:30:27.288313 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:27.288291 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-95637065/e2e-lifecycle-model-95637065-kserve-5fcd5dbb75-8r4b8"] Jul 10 18:30:28.983968 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:28.983932 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c24c01a3-5f5c-4000-afc3-ace373bf337b" path="/var/lib/kubelet/pods/c24c01a3-5f5c-4000-afc3-ace373bf337b/volumes" Jul 10 18:30:51.760087 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:30:51.760051 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:31:05.049401 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:31:05.049365 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:31:19.286128 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:31:19.286049 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:32:07.264907 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:07.264867 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:32:12.822066 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.822037 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl"] Jul 10 18:32:12.822438 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.822376 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24c01a3-5f5c-4000-afc3-ace373bf337b" containerName="main" Jul 10 18:32:12.822438 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.822386 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24c01a3-5f5c-4000-afc3-ace373bf337b" containerName="main" Jul 10 18:32:12.822506 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.822455 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c24c01a3-5f5c-4000-afc3-ace373bf337b" containerName="main" Jul 10 18:32:12.825519 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.825500 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:12.829252 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.829231 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ea8030\"/\"openshift-service-ca.crt\"" Jul 10 18:32:12.829252 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.829246 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ea8030\"/\"default-dockercfg-zc8v9\"" Jul 10 18:32:12.829410 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.829326 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ea8030\"/\"test-model-ea8030-kserve-self-signed-certs\"" Jul 10 18:32:12.829461 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.829448 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ea8030\"/\"kube-root-ca.crt\"" Jul 10 18:32:12.834455 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.834433 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl"] Jul 10 18:32:12.951957 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.951929 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-tmp-dir\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:12.952128 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.952051 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-model-cache\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:12.952128 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.952074 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77gpv\" (UniqueName: \"kubernetes.io/projected/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-kube-api-access-77gpv\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:12.952128 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.952109 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-home\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:12.952128 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.952130 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-dshm\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:12.952282 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:12.952146 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-tls-certs\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.052802 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.052770 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-tmp-dir\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.052973 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.052875 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-model-cache\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.052973 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.052897 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gpv\" (UniqueName: \"kubernetes.io/projected/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-kube-api-access-77gpv\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.052973 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.052926 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-home\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.052973 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.052948 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-dshm\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.052973 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.052970 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-tls-certs\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.053229 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.053198 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-tmp-dir\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.053349 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.053267 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-model-cache\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.053349 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.053314 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-home\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.055153 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.055132 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-dshm\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.055547 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.055527 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-tls-certs\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.061043 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.061021 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77gpv\" (UniqueName: \"kubernetes.io/projected/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-kube-api-access-77gpv\") pod \"test-model-ea8030-kserve-54fb7c4875-4sbvl\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.135162 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.135088 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:13.264580 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.264552 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl"] Jul 10 18:32:13.266520 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:32:13.266495 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod798bc406_774b_4c3a_93f7_7f6b03e2c8ec.slice/crio-54211c074bcd4b4d075786fe47ff2c2ce9250d587da601874ecea3122884c424 WatchSource:0}: Error finding container 54211c074bcd4b4d075786fe47ff2c2ce9250d587da601874ecea3122884c424: Status 404 returned error can't find the container with id 54211c074bcd4b4d075786fe47ff2c2ce9250d587da601874ecea3122884c424 Jul 10 18:32:13.440144 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.440058 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" event={"ID":"798bc406-774b-4c3a-93f7-7f6b03e2c8ec","Type":"ContainerStarted","Data":"221b74564f09702ee5218ff17a2ea34faf8590ad62e5ea379d1d77d239f05a78"} Jul 10 18:32:13.440144 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.440096 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" event={"ID":"798bc406-774b-4c3a-93f7-7f6b03e2c8ec","Type":"ContainerStarted","Data":"54211c074bcd4b4d075786fe47ff2c2ce9250d587da601874ecea3122884c424"} Jul 10 18:32:13.458101 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:13.458052 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" podStartSLOduration=1.458036474 podStartE2EDuration="1.458036474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:32:13.456622883 +0000 UTC m=+1937.003086038" watchObservedRunningTime="2026-07-10 18:32:13.458036474 +0000 UTC m=+1937.004499616" Jul 10 18:32:23.136162 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:23.136122 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:23.136776 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:23.136285 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:23.149087 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:23.149059 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:23.493706 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:23.493620 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:32:23.942737 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:23.942698 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:32:39.524409 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.524375 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj"] Jul 10 18:32:39.528080 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.528057 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.530866 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.530842 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-76cdbe\"/\"openshift-service-ca.crt\"" Jul 10 18:32:39.531850 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.531827 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-76cdbe\"/\"kube-root-ca.crt\"" Jul 10 18:32:39.531945 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.531900 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-76cdbe\"/\"default-dockercfg-mdhn7\"" Jul 10 18:32:39.531945 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.531837 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-76cdbe\"/\"test-model-76cdbe-kserve-self-signed-certs\"" Jul 10 18:32:39.537956 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.537937 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj"] Jul 10 18:32:39.578222 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.578184 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-model-cache\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.578384 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.578265 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a14b53-dc38-4be7-a551-0e2c485a2d63-tls-certs\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.578384 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.578294 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4nxx\" (UniqueName: \"kubernetes.io/projected/c9a14b53-dc38-4be7-a551-0e2c485a2d63-kube-api-access-h4nxx\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.578384 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.578321 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-home\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.578551 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.578403 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-tmp-dir\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.578551 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.578462 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-dshm\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.679519 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.679485 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a14b53-dc38-4be7-a551-0e2c485a2d63-tls-certs\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.679698 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.679526 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4nxx\" (UniqueName: \"kubernetes.io/projected/c9a14b53-dc38-4be7-a551-0e2c485a2d63-kube-api-access-h4nxx\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.679698 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.679546 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-home\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.679698 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.679575 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-tmp-dir\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.679905 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.679762 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-dshm\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.679905 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.679880 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-model-cache\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.680015 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.679997 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-tmp-dir\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.680067 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.680027 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-home\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.680187 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.680167 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-model-cache\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.681952 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.681929 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-dshm\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.682155 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.682137 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a14b53-dc38-4be7-a551-0e2c485a2d63-tls-certs\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.687162 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.687143 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4nxx\" (UniqueName: \"kubernetes.io/projected/c9a14b53-dc38-4be7-a551-0e2c485a2d63-kube-api-access-h4nxx\") pod \"test-model-76cdbe-kserve-b64f4f59-k9lwj\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.839804 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.839769 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:39.974363 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:39.974335 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj"] Jul 10 18:32:39.976890 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:32:39.976858 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9a14b53_dc38_4be7_a551_0e2c485a2d63.slice/crio-06e3b2dfa94472ad5111a8378239bb81e17d065b598d241d1ddb1a263319d678 WatchSource:0}: Error finding container 06e3b2dfa94472ad5111a8378239bb81e17d065b598d241d1ddb1a263319d678: Status 404 returned error can't find the container with id 06e3b2dfa94472ad5111a8378239bb81e17d065b598d241d1ddb1a263319d678 Jul 10 18:32:40.557935 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:40.557899 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" event={"ID":"c9a14b53-dc38-4be7-a551-0e2c485a2d63","Type":"ContainerStarted","Data":"a1034fa6255989498a189fccd32f137ce682d247d9f5be6334641bd1447ec842"} Jul 10 18:32:40.557935 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:40.557941 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" event={"ID":"c9a14b53-dc38-4be7-a551-0e2c485a2d63","Type":"ContainerStarted","Data":"06e3b2dfa94472ad5111a8378239bb81e17d065b598d241d1ddb1a263319d678"} Jul 10 18:32:40.575411 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:40.575361 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" podStartSLOduration=1.575346073 podStartE2EDuration="1.575346073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:32:40.574628823 +0000 UTC m=+1964.121091965" watchObservedRunningTime="2026-07-10 18:32:40.575346073 +0000 UTC m=+1964.121809193" Jul 10 18:32:44.347025 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:44.346990 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:32:49.839980 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:49.839952 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:49.840400 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:49.840157 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:49.852479 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:49.852455 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:50.612912 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:50.612886 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:32:59.441021 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:32:59.440975 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:33:19.828993 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:19.828962 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl"] Jul 10 18:33:20.857888 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:20.857850 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj"] Jul 10 18:33:24.946714 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:24.946686 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:33:34.848603 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:34.848564 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" podUID="798bc406-774b-4c3a-93f7-7f6b03e2c8ec" containerName="main" containerID="cri-o://221b74564f09702ee5218ff17a2ea34faf8590ad62e5ea379d1d77d239f05a78" gracePeriod=45 Jul 10 18:33:35.094234 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.094210 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:33:35.137692 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.137624 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-home\") pod \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " Jul 10 18:33:35.137692 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.137659 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-tls-certs\") pod \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " Jul 10 18:33:35.137692 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.137682 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77gpv\" (UniqueName: \"kubernetes.io/projected/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-kube-api-access-77gpv\") pod \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " Jul 10 18:33:35.137976 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.137709 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-model-cache\") pod \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " Jul 10 18:33:35.137976 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.137830 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-tmp-dir\") pod \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " Jul 10 18:33:35.137976 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.137854 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-dshm\") pod \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\" (UID: \"798bc406-774b-4c3a-93f7-7f6b03e2c8ec\") " Jul 10 18:33:35.137976 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.137883 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-home" (OuterVolumeSpecName: "home") pod "798bc406-774b-4c3a-93f7-7f6b03e2c8ec" (UID: "798bc406-774b-4c3a-93f7-7f6b03e2c8ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:33:35.138190 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.138048 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "798bc406-774b-4c3a-93f7-7f6b03e2c8ec" (UID: "798bc406-774b-4c3a-93f7-7f6b03e2c8ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:33:35.138190 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.138085 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "798bc406-774b-4c3a-93f7-7f6b03e2c8ec" (UID: "798bc406-774b-4c3a-93f7-7f6b03e2c8ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:33:35.138190 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.138173 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-tmp-dir\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:33:35.138359 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.138193 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-home\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:33:35.138359 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.138208 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-model-cache\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:33:35.139716 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.139678 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-kube-api-access-77gpv" (OuterVolumeSpecName: "kube-api-access-77gpv") pod "798bc406-774b-4c3a-93f7-7f6b03e2c8ec" (UID: "798bc406-774b-4c3a-93f7-7f6b03e2c8ec"). InnerVolumeSpecName "kube-api-access-77gpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:33:35.139716 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.139692 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "798bc406-774b-4c3a-93f7-7f6b03e2c8ec" (UID: "798bc406-774b-4c3a-93f7-7f6b03e2c8ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:33:35.139918 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.139853 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-dshm" (OuterVolumeSpecName: "dshm") pod "798bc406-774b-4c3a-93f7-7f6b03e2c8ec" (UID: "798bc406-774b-4c3a-93f7-7f6b03e2c8ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:33:35.239557 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.239533 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-dshm\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:33:35.239557 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.239559 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-tls-certs\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:33:35.239711 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.239569 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77gpv\" (UniqueName: \"kubernetes.io/projected/798bc406-774b-4c3a-93f7-7f6b03e2c8ec-kube-api-access-77gpv\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:33:35.805797 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.805760 2569 generic.go:356] "Generic (PLEG): container finished" podID="798bc406-774b-4c3a-93f7-7f6b03e2c8ec" containerID="221b74564f09702ee5218ff17a2ea34faf8590ad62e5ea379d1d77d239f05a78" exitCode=0 Jul 10 18:33:35.805986 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.805826 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" Jul 10 18:33:35.805986 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.805845 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" event={"ID":"798bc406-774b-4c3a-93f7-7f6b03e2c8ec","Type":"ContainerDied","Data":"221b74564f09702ee5218ff17a2ea34faf8590ad62e5ea379d1d77d239f05a78"} Jul 10 18:33:35.805986 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.805887 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl" event={"ID":"798bc406-774b-4c3a-93f7-7f6b03e2c8ec","Type":"ContainerDied","Data":"54211c074bcd4b4d075786fe47ff2c2ce9250d587da601874ecea3122884c424"} Jul 10 18:33:35.805986 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.805903 2569 scope.go:117] "RemoveContainer" containerID="221b74564f09702ee5218ff17a2ea34faf8590ad62e5ea379d1d77d239f05a78" Jul 10 18:33:35.814352 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.814338 2569 scope.go:117] "RemoveContainer" containerID="221b74564f09702ee5218ff17a2ea34faf8590ad62e5ea379d1d77d239f05a78" Jul 10 18:33:35.814611 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:33:35.814594 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221b74564f09702ee5218ff17a2ea34faf8590ad62e5ea379d1d77d239f05a78\": container with ID starting with 221b74564f09702ee5218ff17a2ea34faf8590ad62e5ea379d1d77d239f05a78 not found: ID does not exist" containerID="221b74564f09702ee5218ff17a2ea34faf8590ad62e5ea379d1d77d239f05a78" Jul 10 18:33:35.814692 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.814618 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221b74564f09702ee5218ff17a2ea34faf8590ad62e5ea379d1d77d239f05a78"} err="failed to get container status \"221b74564f09702ee5218ff17a2ea34faf8590ad62e5ea379d1d77d239f05a78\": rpc error: code = NotFound desc = could not find container \"221b74564f09702ee5218ff17a2ea34faf8590ad62e5ea379d1d77d239f05a78\": container with ID starting with 221b74564f09702ee5218ff17a2ea34faf8590ad62e5ea379d1d77d239f05a78 not found: ID does not exist" Jul 10 18:33:35.825555 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.825532 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl"] Jul 10 18:33:35.828888 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.828869 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ea8030/test-model-ea8030-kserve-54fb7c4875-4sbvl"] Jul 10 18:33:35.878112 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:35.878067 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" podUID="c9a14b53-dc38-4be7-a551-0e2c485a2d63" containerName="main" containerID="cri-o://a1034fa6255989498a189fccd32f137ce682d247d9f5be6334641bd1447ec842" gracePeriod=45 Jul 10 18:33:36.121243 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.121220 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:33:36.147656 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.147637 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4nxx\" (UniqueName: \"kubernetes.io/projected/c9a14b53-dc38-4be7-a551-0e2c485a2d63-kube-api-access-h4nxx\") pod \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " Jul 10 18:33:36.147790 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.147674 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-tmp-dir\") pod \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " Jul 10 18:33:36.147790 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.147702 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-model-cache\") pod \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " Jul 10 18:33:36.147790 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.147746 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a14b53-dc38-4be7-a551-0e2c485a2d63-tls-certs\") pod \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " Jul 10 18:33:36.147979 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.147800 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-home\") pod \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " Jul 10 18:33:36.147979 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.147935 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-dshm\") pod \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\" (UID: \"c9a14b53-dc38-4be7-a551-0e2c485a2d63\") " Jul 10 18:33:36.147979 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.147942 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9a14b53-dc38-4be7-a551-0e2c485a2d63" (UID: "c9a14b53-dc38-4be7-a551-0e2c485a2d63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:33:36.148135 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.148018 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9a14b53-dc38-4be7-a551-0e2c485a2d63" (UID: "c9a14b53-dc38-4be7-a551-0e2c485a2d63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:33:36.148135 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.148070 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-home" (OuterVolumeSpecName: "home") pod "c9a14b53-dc38-4be7-a551-0e2c485a2d63" (UID: "c9a14b53-dc38-4be7-a551-0e2c485a2d63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:33:36.148502 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.148485 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-tmp-dir\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:33:36.148590 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.148506 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-model-cache\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:33:36.148590 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.148519 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-home\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:33:36.149897 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.149873 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a14b53-dc38-4be7-a551-0e2c485a2d63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9a14b53-dc38-4be7-a551-0e2c485a2d63" (UID: "c9a14b53-dc38-4be7-a551-0e2c485a2d63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:33:36.149975 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.149890 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a14b53-dc38-4be7-a551-0e2c485a2d63-kube-api-access-h4nxx" (OuterVolumeSpecName: "kube-api-access-h4nxx") pod "c9a14b53-dc38-4be7-a551-0e2c485a2d63" (UID: "c9a14b53-dc38-4be7-a551-0e2c485a2d63"). InnerVolumeSpecName "kube-api-access-h4nxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:33:36.150236 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.150212 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-dshm" (OuterVolumeSpecName: "dshm") pod "c9a14b53-dc38-4be7-a551-0e2c485a2d63" (UID: "c9a14b53-dc38-4be7-a551-0e2c485a2d63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:33:36.249150 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.249126 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4nxx\" (UniqueName: \"kubernetes.io/projected/c9a14b53-dc38-4be7-a551-0e2c485a2d63-kube-api-access-h4nxx\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:33:36.249236 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.249154 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9a14b53-dc38-4be7-a551-0e2c485a2d63-tls-certs\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:33:36.249236 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.249170 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9a14b53-dc38-4be7-a551-0e2c485a2d63-dshm\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:33:36.812492 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.812457 2569 generic.go:356] "Generic (PLEG): container finished" podID="c9a14b53-dc38-4be7-a551-0e2c485a2d63" containerID="a1034fa6255989498a189fccd32f137ce682d247d9f5be6334641bd1447ec842" exitCode=0 Jul 10 18:33:36.812666 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.812504 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" Jul 10 18:33:36.812666 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.812541 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" event={"ID":"c9a14b53-dc38-4be7-a551-0e2c485a2d63","Type":"ContainerDied","Data":"a1034fa6255989498a189fccd32f137ce682d247d9f5be6334641bd1447ec842"} Jul 10 18:33:36.812666 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.812577 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj" event={"ID":"c9a14b53-dc38-4be7-a551-0e2c485a2d63","Type":"ContainerDied","Data":"06e3b2dfa94472ad5111a8378239bb81e17d065b598d241d1ddb1a263319d678"} Jul 10 18:33:36.812666 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.812596 2569 scope.go:117] "RemoveContainer" containerID="a1034fa6255989498a189fccd32f137ce682d247d9f5be6334641bd1447ec842" Jul 10 18:33:36.821693 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.821676 2569 scope.go:117] "RemoveContainer" containerID="a1034fa6255989498a189fccd32f137ce682d247d9f5be6334641bd1447ec842" Jul 10 18:33:36.821979 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:33:36.821949 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1034fa6255989498a189fccd32f137ce682d247d9f5be6334641bd1447ec842\": container with ID starting with a1034fa6255989498a189fccd32f137ce682d247d9f5be6334641bd1447ec842 not found: ID does not exist" containerID="a1034fa6255989498a189fccd32f137ce682d247d9f5be6334641bd1447ec842" Jul 10 18:33:36.822022 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.821988 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1034fa6255989498a189fccd32f137ce682d247d9f5be6334641bd1447ec842"} err="failed to get container status \"a1034fa6255989498a189fccd32f137ce682d247d9f5be6334641bd1447ec842\": rpc error: code = NotFound desc = could not find container \"a1034fa6255989498a189fccd32f137ce682d247d9f5be6334641bd1447ec842\": container with ID starting with a1034fa6255989498a189fccd32f137ce682d247d9f5be6334641bd1447ec842 not found: ID does not exist" Jul 10 18:33:36.834889 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.834868 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj"] Jul 10 18:33:36.839317 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.839298 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-76cdbe/test-model-76cdbe-kserve-b64f4f59-k9lwj"] Jul 10 18:33:36.983786 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.983763 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="798bc406-774b-4c3a-93f7-7f6b03e2c8ec" path="/var/lib/kubelet/pods/798bc406-774b-4c3a-93f7-7f6b03e2c8ec/volumes" Jul 10 18:33:36.984166 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:33:36.984152 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9a14b53-dc38-4be7-a551-0e2c485a2d63" path="/var/lib/kubelet/pods/c9a14b53-dc38-4be7-a551-0e2c485a2d63/volumes" Jul 10 18:34:08.147534 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:34:08.147498 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:34:19.627413 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:34:19.627332 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:34:29.744084 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:34:29.744051 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:34:46.842015 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:34:46.841985 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:34:57.034224 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:34:57.034189 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:34:57.038777 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:34:57.038754 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:34:57.039793 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:34:57.039774 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:34:57.044142 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:34:57.044125 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:35:00.241843 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:00.241793 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:35:12.245546 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:12.245512 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:35:20.250986 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.250952 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc"] Jul 10 18:35:20.251640 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.251474 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="798bc406-774b-4c3a-93f7-7f6b03e2c8ec" containerName="main" Jul 10 18:35:20.251640 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.251495 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="798bc406-774b-4c3a-93f7-7f6b03e2c8ec" containerName="main" Jul 10 18:35:20.251640 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.251518 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a14b53-dc38-4be7-a551-0e2c485a2d63" containerName="main" Jul 10 18:35:20.251640 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.251528 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a14b53-dc38-4be7-a551-0e2c485a2d63" containerName="main" Jul 10 18:35:20.251640 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.251623 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="798bc406-774b-4c3a-93f7-7f6b03e2c8ec" containerName="main" Jul 10 18:35:20.251640 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.251641 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a14b53-dc38-4be7-a551-0e2c485a2d63" containerName="main" Jul 10 18:35:20.254825 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.254789 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.257233 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.257210 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-48683a\"/\"openshift-service-ca.crt\"" Jul 10 18:35:20.258278 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.258262 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48683a\"/\"default-dockercfg-ghzjg\"" Jul 10 18:35:20.258370 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.258352 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-48683a\"/\"kube-root-ca.crt\"" Jul 10 18:35:20.258488 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.258476 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48683a\"/\"auth-test-model-48683a-kserve-self-signed-certs\"" Jul 10 18:35:20.269271 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.269242 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc"] Jul 10 18:35:20.383766 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.383730 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-tmp-dir\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.383983 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.383785 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-model-cache\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.383983 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.383836 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjcbf\" (UniqueName: \"kubernetes.io/projected/64adc90d-acf5-429e-a93c-11a97e5dbb33-kube-api-access-jjcbf\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.383983 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.383877 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64adc90d-acf5-429e-a93c-11a97e5dbb33-tls-certs\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.383983 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.383895 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-home\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.383983 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.383923 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-dshm\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.484656 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.484619 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64adc90d-acf5-429e-a93c-11a97e5dbb33-tls-certs\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.484656 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.484659 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-home\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.484966 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.484681 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-dshm\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.484966 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.484748 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-tmp-dir\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.484966 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.484790 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-model-cache\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.484966 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.484846 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjcbf\" (UniqueName: \"kubernetes.io/projected/64adc90d-acf5-429e-a93c-11a97e5dbb33-kube-api-access-jjcbf\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.485199 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.485169 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-home\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.485270 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.485191 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-tmp-dir\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.485324 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.485275 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-model-cache\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.487137 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.487116 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-dshm\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.487390 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.487375 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64adc90d-acf5-429e-a93c-11a97e5dbb33-tls-certs\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.493240 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.493217 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjcbf\" (UniqueName: \"kubernetes.io/projected/64adc90d-acf5-429e-a93c-11a97e5dbb33-kube-api-access-jjcbf\") pod \"auth-test-model-48683a-kserve-5d7df56c5-zfvvc\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.564336 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.564306 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:20.713464 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:20.713425 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc"] Jul 10 18:35:20.715855 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:35:20.715823 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64adc90d_acf5_429e_a93c_11a97e5dbb33.slice/crio-9001fe5c5ecef0f85f2bc67356515ce1daee94f7fe595c69d147ce3840d65ae0 WatchSource:0}: Error finding container 9001fe5c5ecef0f85f2bc67356515ce1daee94f7fe595c69d147ce3840d65ae0: Status 404 returned error can't find the container with id 9001fe5c5ecef0f85f2bc67356515ce1daee94f7fe595c69d147ce3840d65ae0 Jul 10 18:35:21.292547 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:21.292499 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" event={"ID":"64adc90d-acf5-429e-a93c-11a97e5dbb33","Type":"ContainerStarted","Data":"6c537c150f91fd229fc30f4f4519b2161c65ba79d3f8bfdf1fc77089c9b29dad"} Jul 10 18:35:21.292547 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:21.292554 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" event={"ID":"64adc90d-acf5-429e-a93c-11a97e5dbb33","Type":"ContainerStarted","Data":"9001fe5c5ecef0f85f2bc67356515ce1daee94f7fe595c69d147ce3840d65ae0"} Jul 10 18:35:21.311917 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:21.311856 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" podStartSLOduration=1.311839789 podStartE2EDuration="1.311839789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:35:21.311618272 +0000 UTC m=+2124.858081415" watchObservedRunningTime="2026-07-10 18:35:21.311839789 +0000 UTC m=+2124.858302932" Jul 10 18:35:24.547760 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:24.547719 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:35:30.565500 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:30.565457 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:30.565500 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:30.565511 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:30.578518 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:30.578484 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:31.349126 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:31.349093 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:35:57.144905 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.144874 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk"] Jul 10 18:35:57.148706 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.148685 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.151777 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.151758 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35792e\"/\"kube-root-ca.crt\"" Jul 10 18:35:57.152771 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.152754 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35792e\"/\"openshift-service-ca.crt\"" Jul 10 18:35:57.152877 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.152780 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35792e\"/\"auth-test-model-35792e-kserve-self-signed-certs\"" Jul 10 18:35:57.152877 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.152830 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35792e\"/\"default-dockercfg-tqqqc\"" Jul 10 18:35:57.161889 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.161870 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk"] Jul 10 18:35:57.218900 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.218866 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1ade0a-7f22-42ce-8710-7df9e5a25345-tls-certs\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.218900 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.218901 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hvkn\" (UniqueName: \"kubernetes.io/projected/7a1ade0a-7f22-42ce-8710-7df9e5a25345-kube-api-access-2hvkn\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.219089 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.218963 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-tmp-dir\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.219089 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.218988 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-home\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.219089 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.219035 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-dshm\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.219201 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.219106 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-model-cache\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.320541 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.320509 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1ade0a-7f22-42ce-8710-7df9e5a25345-tls-certs\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.320541 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.320544 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hvkn\" (UniqueName: \"kubernetes.io/projected/7a1ade0a-7f22-42ce-8710-7df9e5a25345-kube-api-access-2hvkn\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.320796 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.320576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-tmp-dir\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.320796 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.320593 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-home\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.320796 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.320644 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-dshm\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.320796 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.320696 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-model-cache\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.321055 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.321019 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-tmp-dir\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.321175 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.321151 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-model-cache\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.321239 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.321155 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-home\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.322916 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.322895 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-dshm\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.323181 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.323162 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1ade0a-7f22-42ce-8710-7df9e5a25345-tls-certs\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.333009 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.332987 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hvkn\" (UniqueName: \"kubernetes.io/projected/7a1ade0a-7f22-42ce-8710-7df9e5a25345-kube-api-access-2hvkn\") pod \"auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.457207 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.457124 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:35:57.593298 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:57.593274 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk"] Jul 10 18:35:57.595461 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:35:57.595435 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a1ade0a_7f22_42ce_8710_7df9e5a25345.slice/crio-f1c0f0254260b63a085465afa7c172bc3e5b870ca63d7d51ad68a5be9642a314 WatchSource:0}: Error finding container f1c0f0254260b63a085465afa7c172bc3e5b870ca63d7d51ad68a5be9642a314: Status 404 returned error can't find the container with id f1c0f0254260b63a085465afa7c172bc3e5b870ca63d7d51ad68a5be9642a314 Jul 10 18:35:58.459656 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:58.459603 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" event={"ID":"7a1ade0a-7f22-42ce-8710-7df9e5a25345","Type":"ContainerStarted","Data":"9b8bb07c70fe59b7efd5c205e1e698f4aca510dd9c1f7fdfda97d4b88532dce4"} Jul 10 18:35:58.459656 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:58.459650 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" event={"ID":"7a1ade0a-7f22-42ce-8710-7df9e5a25345","Type":"ContainerStarted","Data":"f1c0f0254260b63a085465afa7c172bc3e5b870ca63d7d51ad68a5be9642a314"} Jul 10 18:35:58.489834 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:35:58.489766 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" podStartSLOduration=1.489750462 podStartE2EDuration="1.489750462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:35:58.488456051 +0000 UTC m=+2162.034919204" watchObservedRunningTime="2026-07-10 18:35:58.489750462 +0000 UTC m=+2162.036213582" Jul 10 18:36:01.256273 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:36:01.256231 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:36:07.457367 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:36:07.457332 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:36:07.457970 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:36:07.457389 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:36:07.471224 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:36:07.471200 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:36:07.511657 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:36:07.511631 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:36:13.267392 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:36:13.267355 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:36:31.359303 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:36:31.359267 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:36:48.964803 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:36:48.964772 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:37:07.158003 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:37:07.157970 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:37:28.153044 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:37:28.152955 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:37:45.860408 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:37:45.860376 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:38:01.570176 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:01.570141 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc"] Jul 10 18:38:02.528560 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:02.528508 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk"] Jul 10 18:38:05.648222 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:05.648185 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:38:16.591175 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.591118 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" podUID="64adc90d-acf5-429e-a93c-11a97e5dbb33" containerName="main" containerID="cri-o://6c537c150f91fd229fc30f4f4519b2161c65ba79d3f8bfdf1fc77089c9b29dad" gracePeriod=45 Jul 10 18:38:16.840271 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.840241 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:38:16.870447 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.870383 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjcbf\" (UniqueName: \"kubernetes.io/projected/64adc90d-acf5-429e-a93c-11a97e5dbb33-kube-api-access-jjcbf\") pod \"64adc90d-acf5-429e-a93c-11a97e5dbb33\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " Jul 10 18:38:16.870593 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.870463 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-model-cache\") pod \"64adc90d-acf5-429e-a93c-11a97e5dbb33\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " Jul 10 18:38:16.870593 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.870510 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-dshm\") pod \"64adc90d-acf5-429e-a93c-11a97e5dbb33\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " Jul 10 18:38:16.870593 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.870552 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-home\") pod \"64adc90d-acf5-429e-a93c-11a97e5dbb33\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " Jul 10 18:38:16.870593 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.870586 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-tmp-dir\") pod \"64adc90d-acf5-429e-a93c-11a97e5dbb33\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " Jul 10 18:38:16.870801 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.870663 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64adc90d-acf5-429e-a93c-11a97e5dbb33-tls-certs\") pod \"64adc90d-acf5-429e-a93c-11a97e5dbb33\" (UID: \"64adc90d-acf5-429e-a93c-11a97e5dbb33\") " Jul 10 18:38:16.870801 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.870735 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-model-cache" (OuterVolumeSpecName: "model-cache") pod "64adc90d-acf5-429e-a93c-11a97e5dbb33" (UID: "64adc90d-acf5-429e-a93c-11a97e5dbb33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:16.870937 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.870857 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-home" (OuterVolumeSpecName: "home") pod "64adc90d-acf5-429e-a93c-11a97e5dbb33" (UID: "64adc90d-acf5-429e-a93c-11a97e5dbb33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:16.870937 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.870867 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64adc90d-acf5-429e-a93c-11a97e5dbb33" (UID: "64adc90d-acf5-429e-a93c-11a97e5dbb33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:16.871046 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.871011 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-model-cache\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:38:16.871046 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.871030 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-home\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:38:16.871046 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.871044 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-tmp-dir\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:38:16.872651 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.872621 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-dshm" (OuterVolumeSpecName: "dshm") pod "64adc90d-acf5-429e-a93c-11a97e5dbb33" (UID: "64adc90d-acf5-429e-a93c-11a97e5dbb33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:16.872758 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.872682 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64adc90d-acf5-429e-a93c-11a97e5dbb33-kube-api-access-jjcbf" (OuterVolumeSpecName: "kube-api-access-jjcbf") pod "64adc90d-acf5-429e-a93c-11a97e5dbb33" (UID: "64adc90d-acf5-429e-a93c-11a97e5dbb33"). InnerVolumeSpecName "kube-api-access-jjcbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:38:16.873097 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.873068 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64adc90d-acf5-429e-a93c-11a97e5dbb33-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64adc90d-acf5-429e-a93c-11a97e5dbb33" (UID: "64adc90d-acf5-429e-a93c-11a97e5dbb33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:38:16.971822 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.971788 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64adc90d-acf5-429e-a93c-11a97e5dbb33-dshm\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:38:16.971977 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.971836 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64adc90d-acf5-429e-a93c-11a97e5dbb33-tls-certs\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:38:16.971977 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:16.971849 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjcbf\" (UniqueName: \"kubernetes.io/projected/64adc90d-acf5-429e-a93c-11a97e5dbb33-kube-api-access-jjcbf\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:38:17.093675 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.093641 2569 generic.go:356] "Generic (PLEG): container finished" podID="64adc90d-acf5-429e-a93c-11a97e5dbb33" containerID="6c537c150f91fd229fc30f4f4519b2161c65ba79d3f8bfdf1fc77089c9b29dad" exitCode=0 Jul 10 18:38:17.093868 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.093682 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" Jul 10 18:38:17.093868 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.093690 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" event={"ID":"64adc90d-acf5-429e-a93c-11a97e5dbb33","Type":"ContainerDied","Data":"6c537c150f91fd229fc30f4f4519b2161c65ba79d3f8bfdf1fc77089c9b29dad"} Jul 10 18:38:17.093868 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.093728 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc" event={"ID":"64adc90d-acf5-429e-a93c-11a97e5dbb33","Type":"ContainerDied","Data":"9001fe5c5ecef0f85f2bc67356515ce1daee94f7fe595c69d147ce3840d65ae0"} Jul 10 18:38:17.093868 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.093746 2569 scope.go:117] "RemoveContainer" containerID="6c537c150f91fd229fc30f4f4519b2161c65ba79d3f8bfdf1fc77089c9b29dad" Jul 10 18:38:17.104516 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.104431 2569 scope.go:117] "RemoveContainer" containerID="6c537c150f91fd229fc30f4f4519b2161c65ba79d3f8bfdf1fc77089c9b29dad" Jul 10 18:38:17.104834 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:38:17.104792 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c537c150f91fd229fc30f4f4519b2161c65ba79d3f8bfdf1fc77089c9b29dad\": container with ID starting with 6c537c150f91fd229fc30f4f4519b2161c65ba79d3f8bfdf1fc77089c9b29dad not found: ID does not exist" containerID="6c537c150f91fd229fc30f4f4519b2161c65ba79d3f8bfdf1fc77089c9b29dad" Jul 10 18:38:17.104928 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.104842 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c537c150f91fd229fc30f4f4519b2161c65ba79d3f8bfdf1fc77089c9b29dad"} err="failed to get container status \"6c537c150f91fd229fc30f4f4519b2161c65ba79d3f8bfdf1fc77089c9b29dad\": rpc error: code = NotFound desc = could not find container \"6c537c150f91fd229fc30f4f4519b2161c65ba79d3f8bfdf1fc77089c9b29dad\": container with ID starting with 6c537c150f91fd229fc30f4f4519b2161c65ba79d3f8bfdf1fc77089c9b29dad not found: ID does not exist" Jul 10 18:38:17.111882 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.111838 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc"] Jul 10 18:38:17.115872 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.115854 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-48683a/auth-test-model-48683a-kserve-5d7df56c5-zfvvc"] Jul 10 18:38:17.553426 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.553387 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" podUID="7a1ade0a-7f22-42ce-8710-7df9e5a25345" containerName="main" containerID="cri-o://9b8bb07c70fe59b7efd5c205e1e698f4aca510dd9c1f7fdfda97d4b88532dce4" gracePeriod=45 Jul 10 18:38:17.798717 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.798692 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:38:17.879744 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.879646 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hvkn\" (UniqueName: \"kubernetes.io/projected/7a1ade0a-7f22-42ce-8710-7df9e5a25345-kube-api-access-2hvkn\") pod \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " Jul 10 18:38:17.879744 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.879690 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-dshm\") pod \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " Jul 10 18:38:17.879744 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.879734 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-model-cache\") pod \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " Jul 10 18:38:17.880070 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.879772 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1ade0a-7f22-42ce-8710-7df9e5a25345-tls-certs\") pod \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " Jul 10 18:38:17.880070 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.879846 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-tmp-dir\") pod \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " Jul 10 18:38:17.880070 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.879860 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-home\") pod \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\" (UID: \"7a1ade0a-7f22-42ce-8710-7df9e5a25345\") " Jul 10 18:38:17.880239 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.880114 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a1ade0a-7f22-42ce-8710-7df9e5a25345" (UID: "7a1ade0a-7f22-42ce-8710-7df9e5a25345"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:17.880239 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.880097 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a1ade0a-7f22-42ce-8710-7df9e5a25345" (UID: "7a1ade0a-7f22-42ce-8710-7df9e5a25345"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:17.880239 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.880165 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-home" (OuterVolumeSpecName: "home") pod "7a1ade0a-7f22-42ce-8710-7df9e5a25345" (UID: "7a1ade0a-7f22-42ce-8710-7df9e5a25345"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:17.882067 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.882044 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-dshm" (OuterVolumeSpecName: "dshm") pod "7a1ade0a-7f22-42ce-8710-7df9e5a25345" (UID: "7a1ade0a-7f22-42ce-8710-7df9e5a25345"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:17.882361 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.882338 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1ade0a-7f22-42ce-8710-7df9e5a25345-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a1ade0a-7f22-42ce-8710-7df9e5a25345" (UID: "7a1ade0a-7f22-42ce-8710-7df9e5a25345"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:38:17.882411 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.882346 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1ade0a-7f22-42ce-8710-7df9e5a25345-kube-api-access-2hvkn" (OuterVolumeSpecName: "kube-api-access-2hvkn") pod "7a1ade0a-7f22-42ce-8710-7df9e5a25345" (UID: "7a1ade0a-7f22-42ce-8710-7df9e5a25345"). InnerVolumeSpecName "kube-api-access-2hvkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:38:17.980779 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.980748 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1ade0a-7f22-42ce-8710-7df9e5a25345-tls-certs\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:38:17.980779 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.980773 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-tmp-dir\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:38:17.980779 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.980784 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-home\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:38:17.981008 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.980792 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hvkn\" (UniqueName: \"kubernetes.io/projected/7a1ade0a-7f22-42ce-8710-7df9e5a25345-kube-api-access-2hvkn\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:38:17.981008 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.980801 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-dshm\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:38:17.981008 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:17.980827 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1ade0a-7f22-42ce-8710-7df9e5a25345-model-cache\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:38:18.100522 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:18.100482 2569 generic.go:356] "Generic (PLEG): container finished" podID="7a1ade0a-7f22-42ce-8710-7df9e5a25345" containerID="9b8bb07c70fe59b7efd5c205e1e698f4aca510dd9c1f7fdfda97d4b88532dce4" exitCode=0 Jul 10 18:38:18.100522 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:18.100527 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" Jul 10 18:38:18.100739 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:18.100528 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" event={"ID":"7a1ade0a-7f22-42ce-8710-7df9e5a25345","Type":"ContainerDied","Data":"9b8bb07c70fe59b7efd5c205e1e698f4aca510dd9c1f7fdfda97d4b88532dce4"} Jul 10 18:38:18.100739 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:18.100566 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk" event={"ID":"7a1ade0a-7f22-42ce-8710-7df9e5a25345","Type":"ContainerDied","Data":"f1c0f0254260b63a085465afa7c172bc3e5b870ca63d7d51ad68a5be9642a314"} Jul 10 18:38:18.100739 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:18.100588 2569 scope.go:117] "RemoveContainer" containerID="9b8bb07c70fe59b7efd5c205e1e698f4aca510dd9c1f7fdfda97d4b88532dce4" Jul 10 18:38:18.109189 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:18.109165 2569 scope.go:117] "RemoveContainer" containerID="9b8bb07c70fe59b7efd5c205e1e698f4aca510dd9c1f7fdfda97d4b88532dce4" Jul 10 18:38:18.109425 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:38:18.109406 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8bb07c70fe59b7efd5c205e1e698f4aca510dd9c1f7fdfda97d4b88532dce4\": container with ID starting with 9b8bb07c70fe59b7efd5c205e1e698f4aca510dd9c1f7fdfda97d4b88532dce4 not found: ID does not exist" containerID="9b8bb07c70fe59b7efd5c205e1e698f4aca510dd9c1f7fdfda97d4b88532dce4" Jul 10 18:38:18.109489 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:18.109433 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8bb07c70fe59b7efd5c205e1e698f4aca510dd9c1f7fdfda97d4b88532dce4"} err="failed to get container status \"9b8bb07c70fe59b7efd5c205e1e698f4aca510dd9c1f7fdfda97d4b88532dce4\": rpc error: code = NotFound desc = could not find container \"9b8bb07c70fe59b7efd5c205e1e698f4aca510dd9c1f7fdfda97d4b88532dce4\": container with ID starting with 9b8bb07c70fe59b7efd5c205e1e698f4aca510dd9c1f7fdfda97d4b88532dce4 not found: ID does not exist" Jul 10 18:38:18.120602 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:18.120578 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk"] Jul 10 18:38:18.124319 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:18.124301 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/auth-test-model-35792e-kserve-5c5c7db4bc-cjxmk"] Jul 10 18:38:18.984528 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:18.984484 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64adc90d-acf5-429e-a93c-11a97e5dbb33" path="/var/lib/kubelet/pods/64adc90d-acf5-429e-a93c-11a97e5dbb33/volumes" Jul 10 18:38:18.985098 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:18.985078 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a1ade0a-7f22-42ce-8710-7df9e5a25345" path="/var/lib/kubelet/pods/7a1ade0a-7f22-42ce-8710-7df9e5a25345/volumes" Jul 10 18:38:21.648269 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:21.648229 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:38:33.158771 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.158739 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4"] Jul 10 18:38:33.159230 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.159078 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a1ade0a-7f22-42ce-8710-7df9e5a25345" containerName="main" Jul 10 18:38:33.159230 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.159090 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a1ade0a-7f22-42ce-8710-7df9e5a25345" containerName="main" Jul 10 18:38:33.159230 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.159119 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64adc90d-acf5-429e-a93c-11a97e5dbb33" containerName="main" Jul 10 18:38:33.159230 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.159124 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="64adc90d-acf5-429e-a93c-11a97e5dbb33" containerName="main" Jul 10 18:38:33.159230 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.159175 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a1ade0a-7f22-42ce-8710-7df9e5a25345" containerName="main" Jul 10 18:38:33.159230 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.159186 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="64adc90d-acf5-429e-a93c-11a97e5dbb33" containerName="main" Jul 10 18:38:33.162062 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.162042 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.165001 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.164981 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35792e\"/\"kube-root-ca.crt\"" Jul 10 18:38:33.166015 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.165997 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35792e\"/\"sub-test-model-35792e-kserve-self-signed-certs\"" Jul 10 18:38:33.166095 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.166035 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35792e\"/\"default-dockercfg-tqqqc\"" Jul 10 18:38:33.166157 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.166139 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35792e\"/\"openshift-service-ca.crt\"" Jul 10 18:38:33.173210 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.173189 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4"] Jul 10 18:38:33.328945 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.328895 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-home\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.328945 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.328959 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-model-cache\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.329210 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.329076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slm5w\" (UniqueName: \"kubernetes.io/projected/60d7cfb1-c462-46e7-be9e-86fd78af32b5-kube-api-access-slm5w\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.329210 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.329129 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-tmp-dir\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.329210 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.329186 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60d7cfb1-c462-46e7-be9e-86fd78af32b5-tls-certs\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.329326 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.329236 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-dshm\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.429902 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.429791 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-tmp-dir\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.429902 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.429852 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60d7cfb1-c462-46e7-be9e-86fd78af32b5-tls-certs\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.429902 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.429886 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-dshm\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.430176 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.429913 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-home\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.430176 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.429936 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-model-cache\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.430176 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.429989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slm5w\" (UniqueName: \"kubernetes.io/projected/60d7cfb1-c462-46e7-be9e-86fd78af32b5-kube-api-access-slm5w\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.430342 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.430271 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-tmp-dir\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.430342 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.430303 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-home\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.430443 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.430426 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-model-cache\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.432298 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.432270 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-dshm\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.432507 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.432488 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60d7cfb1-c462-46e7-be9e-86fd78af32b5-tls-certs\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.438855 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.438833 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slm5w\" (UniqueName: \"kubernetes.io/projected/60d7cfb1-c462-46e7-be9e-86fd78af32b5-kube-api-access-slm5w\") pod \"sub-test-model-35792e-kserve-555976d97-f7xl4\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.472017 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.471974 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:33.603597 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:33.603573 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4"] Jul 10 18:38:33.605801 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:38:33.605776 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60d7cfb1_c462_46e7_be9e_86fd78af32b5.slice/crio-db0d985bf5a86b0fd03e5be3ec821358e799851051ae7df7f70d04171f389c5f WatchSource:0}: Error finding container db0d985bf5a86b0fd03e5be3ec821358e799851051ae7df7f70d04171f389c5f: Status 404 returned error can't find the container with id db0d985bf5a86b0fd03e5be3ec821358e799851051ae7df7f70d04171f389c5f Jul 10 18:38:34.176823 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:34.176766 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" event={"ID":"60d7cfb1-c462-46e7-be9e-86fd78af32b5","Type":"ContainerStarted","Data":"ae650b5f8f9cae1711b762f39ae02d5cdb11b6e81bc00cac3d574d51af3cf649"} Jul 10 18:38:34.176823 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:34.176831 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" event={"ID":"60d7cfb1-c462-46e7-be9e-86fd78af32b5","Type":"ContainerStarted","Data":"db0d985bf5a86b0fd03e5be3ec821358e799851051ae7df7f70d04171f389c5f"} Jul 10 18:38:34.197709 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:34.197633 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" podStartSLOduration=1.19761064 podStartE2EDuration="1.19761064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:38:34.19605988 +0000 UTC m=+2317.742523023" watchObservedRunningTime="2026-07-10 18:38:34.19761064 +0000 UTC m=+2317.744073782" Jul 10 18:38:36.745299 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:36.745258 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:38:43.472909 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:43.472862 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:43.472909 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:43.472916 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:43.485232 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:43.485199 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:38:44.233148 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:38:44.233119 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:39:04.158001 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:04.157969 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:39:10.934343 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:10.934307 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4"] Jul 10 18:39:23.552196 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:23.552156 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:39:25.954081 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:25.954014 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" podUID="60d7cfb1-c462-46e7-be9e-86fd78af32b5" containerName="main" containerID="cri-o://ae650b5f8f9cae1711b762f39ae02d5cdb11b6e81bc00cac3d574d51af3cf649" gracePeriod=45 Jul 10 18:39:26.219034 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.218919 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:39:26.319505 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.319445 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-home\") pod \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " Jul 10 18:39:26.319505 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.319513 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-model-cache\") pod \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " Jul 10 18:39:26.319914 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.319567 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60d7cfb1-c462-46e7-be9e-86fd78af32b5-tls-certs\") pod \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " Jul 10 18:39:26.319914 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.319610 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slm5w\" (UniqueName: \"kubernetes.io/projected/60d7cfb1-c462-46e7-be9e-86fd78af32b5-kube-api-access-slm5w\") pod \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " Jul 10 18:39:26.319914 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.319636 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-dshm\") pod \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " Jul 10 18:39:26.319914 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.319854 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-tmp-dir\") pod \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\" (UID: \"60d7cfb1-c462-46e7-be9e-86fd78af32b5\") " Jul 10 18:39:26.319914 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.319891 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "60d7cfb1-c462-46e7-be9e-86fd78af32b5" (UID: "60d7cfb1-c462-46e7-be9e-86fd78af32b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:39:26.320157 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.319903 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-home" (OuterVolumeSpecName: "home") pod "60d7cfb1-c462-46e7-be9e-86fd78af32b5" (UID: "60d7cfb1-c462-46e7-be9e-86fd78af32b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:39:26.320214 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.320171 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60d7cfb1-c462-46e7-be9e-86fd78af32b5" (UID: "60d7cfb1-c462-46e7-be9e-86fd78af32b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:39:26.320392 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.320368 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-tmp-dir\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:39:26.320392 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.320392 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-home\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:39:26.320580 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.320401 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-model-cache\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:39:26.322270 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.322206 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-dshm" (OuterVolumeSpecName: "dshm") pod "60d7cfb1-c462-46e7-be9e-86fd78af32b5" (UID: "60d7cfb1-c462-46e7-be9e-86fd78af32b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:39:26.322425 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.322336 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d7cfb1-c462-46e7-be9e-86fd78af32b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60d7cfb1-c462-46e7-be9e-86fd78af32b5" (UID: "60d7cfb1-c462-46e7-be9e-86fd78af32b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:39:26.322425 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.322362 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d7cfb1-c462-46e7-be9e-86fd78af32b5-kube-api-access-slm5w" (OuterVolumeSpecName: "kube-api-access-slm5w") pod "60d7cfb1-c462-46e7-be9e-86fd78af32b5" (UID: "60d7cfb1-c462-46e7-be9e-86fd78af32b5"). InnerVolumeSpecName "kube-api-access-slm5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:39:26.420342 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.420290 2569 generic.go:356] "Generic (PLEG): container finished" podID="60d7cfb1-c462-46e7-be9e-86fd78af32b5" containerID="ae650b5f8f9cae1711b762f39ae02d5cdb11b6e81bc00cac3d574d51af3cf649" exitCode=0 Jul 10 18:39:26.420342 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.420352 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" Jul 10 18:39:26.420675 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.420378 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" event={"ID":"60d7cfb1-c462-46e7-be9e-86fd78af32b5","Type":"ContainerDied","Data":"ae650b5f8f9cae1711b762f39ae02d5cdb11b6e81bc00cac3d574d51af3cf649"} Jul 10 18:39:26.420675 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.420434 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4" event={"ID":"60d7cfb1-c462-46e7-be9e-86fd78af32b5","Type":"ContainerDied","Data":"db0d985bf5a86b0fd03e5be3ec821358e799851051ae7df7f70d04171f389c5f"} Jul 10 18:39:26.420675 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.420456 2569 scope.go:117] "RemoveContainer" containerID="ae650b5f8f9cae1711b762f39ae02d5cdb11b6e81bc00cac3d574d51af3cf649" Jul 10 18:39:26.421247 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.421212 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60d7cfb1-c462-46e7-be9e-86fd78af32b5-tls-certs\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:39:26.421247 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.421244 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slm5w\" (UniqueName: \"kubernetes.io/projected/60d7cfb1-c462-46e7-be9e-86fd78af32b5-kube-api-access-slm5w\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:39:26.421503 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.421295 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60d7cfb1-c462-46e7-be9e-86fd78af32b5-dshm\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:39:26.432402 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.432378 2569 scope.go:117] "RemoveContainer" containerID="ae650b5f8f9cae1711b762f39ae02d5cdb11b6e81bc00cac3d574d51af3cf649" Jul 10 18:39:26.432857 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:39:26.432829 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae650b5f8f9cae1711b762f39ae02d5cdb11b6e81bc00cac3d574d51af3cf649\": container with ID starting with ae650b5f8f9cae1711b762f39ae02d5cdb11b6e81bc00cac3d574d51af3cf649 not found: ID does not exist" containerID="ae650b5f8f9cae1711b762f39ae02d5cdb11b6e81bc00cac3d574d51af3cf649" Jul 10 18:39:26.432912 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.432872 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae650b5f8f9cae1711b762f39ae02d5cdb11b6e81bc00cac3d574d51af3cf649"} err="failed to get container status \"ae650b5f8f9cae1711b762f39ae02d5cdb11b6e81bc00cac3d574d51af3cf649\": rpc error: code = NotFound desc = could not find container \"ae650b5f8f9cae1711b762f39ae02d5cdb11b6e81bc00cac3d574d51af3cf649\": container with ID starting with ae650b5f8f9cae1711b762f39ae02d5cdb11b6e81bc00cac3d574d51af3cf649 not found: ID does not exist" Jul 10 18:39:26.444371 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.444323 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4"] Jul 10 18:39:26.448802 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.448755 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/sub-test-model-35792e-kserve-555976d97-f7xl4"] Jul 10 18:39:26.984671 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:26.984626 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60d7cfb1-c462-46e7-be9e-86fd78af32b5" path="/var/lib/kubelet/pods/60d7cfb1-c462-46e7-be9e-86fd78af32b5/volumes" Jul 10 18:39:28.048848 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.048793 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf"] Jul 10 18:39:28.049237 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.049193 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d7cfb1-c462-46e7-be9e-86fd78af32b5" containerName="main" Jul 10 18:39:28.049237 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.049205 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d7cfb1-c462-46e7-be9e-86fd78af32b5" containerName="main" Jul 10 18:39:28.049310 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.049266 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="60d7cfb1-c462-46e7-be9e-86fd78af32b5" containerName="main" Jul 10 18:39:28.053344 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.053324 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.056372 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.056352 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48683a\"/\"default-dockercfg-ghzjg\"" Jul 10 18:39:28.057381 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.057366 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48683a\"/\"rate-test-model-48683a-kserve-self-signed-certs\"" Jul 10 18:39:28.057482 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.057383 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-48683a\"/\"kube-root-ca.crt\"" Jul 10 18:39:28.066508 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.066468 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-48683a\"/\"openshift-service-ca.crt\"" Jul 10 18:39:28.068387 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.068368 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf"] Jul 10 18:39:28.138007 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.137980 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-model-cache\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.138234 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.138025 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc89k\" (UniqueName: \"kubernetes.io/projected/f38284ab-e4a5-481e-b3b8-de4513360ebd-kube-api-access-jc89k\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.138234 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.138062 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-tmp-dir\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.138234 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.138115 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-home\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.138234 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.138152 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-dshm\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.138234 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.138166 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38284ab-e4a5-481e-b3b8-de4513360ebd-tls-certs\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.239476 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.239442 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-model-cache\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.239476 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.239486 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc89k\" (UniqueName: \"kubernetes.io/projected/f38284ab-e4a5-481e-b3b8-de4513360ebd-kube-api-access-jc89k\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.239736 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.239510 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-tmp-dir\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.239736 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.239543 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-home\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.239736 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.239560 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-dshm\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.239736 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.239574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38284ab-e4a5-481e-b3b8-de4513360ebd-tls-certs\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.240118 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.240092 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-model-cache\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.240210 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.240121 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-home\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.240210 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.240194 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-tmp-dir\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.242007 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.241987 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-dshm\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.242181 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.242166 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38284ab-e4a5-481e-b3b8-de4513360ebd-tls-certs\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.247635 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.247615 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc89k\" (UniqueName: \"kubernetes.io/projected/f38284ab-e4a5-481e-b3b8-de4513360ebd-kube-api-access-jc89k\") pod \"rate-test-model-48683a-kserve-594f9cb96c-kxxcf\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.363014 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.362979 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:28.493522 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:28.493489 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf"] Jul 10 18:39:28.496668 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:39:28.496641 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf38284ab_e4a5_481e_b3b8_de4513360ebd.slice/crio-1b60af8f75237e91da675d1d6ea2982bb2d23a90db1f886be62dd45922010214 WatchSource:0}: Error finding container 1b60af8f75237e91da675d1d6ea2982bb2d23a90db1f886be62dd45922010214: Status 404 returned error can't find the container with id 1b60af8f75237e91da675d1d6ea2982bb2d23a90db1f886be62dd45922010214 Jul 10 18:39:29.437626 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:29.437578 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" event={"ID":"f38284ab-e4a5-481e-b3b8-de4513360ebd","Type":"ContainerStarted","Data":"8d0ecadaacbc71b82391e321ecac410098cc43efd3130fe75168430d002c73d1"} Jul 10 18:39:29.438127 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:29.437644 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" event={"ID":"f38284ab-e4a5-481e-b3b8-de4513360ebd","Type":"ContainerStarted","Data":"1b60af8f75237e91da675d1d6ea2982bb2d23a90db1f886be62dd45922010214"} Jul 10 18:39:29.455699 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:29.455639 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" podStartSLOduration=1.45562124 podStartE2EDuration="1.45562124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:29.453861614 +0000 UTC m=+2373.000324770" watchObservedRunningTime="2026-07-10 18:39:29.45562124 +0000 UTC m=+2373.002084381" Jul 10 18:39:33.445649 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:33.445612 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:39:38.363731 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:38.363685 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:38.364370 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:38.363756 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:38.376453 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:38.376415 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:38.493530 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:38.493498 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:39:49.748424 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.748388 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f"] Jul 10 18:39:49.751688 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.751667 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.755084 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.755052 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35792e\"/\"openshift-service-ca.crt\"" Jul 10 18:39:49.755084 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.755080 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35792e\"/\"default-dockercfg-tqqqc\"" Jul 10 18:39:49.755268 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.755094 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35792e\"/\"kube-root-ca.crt\"" Jul 10 18:39:49.755268 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.755114 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35792e\"/\"rate-test-model-35792e-kserve-self-signed-certs\"" Jul 10 18:39:49.761668 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.761647 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f"] Jul 10 18:39:49.842483 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.842446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-home\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.842676 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.842521 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7t7g\" (UniqueName: \"kubernetes.io/projected/e5250dc2-7ac2-4868-8654-181200be07ce-kube-api-access-h7t7g\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.842676 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.842572 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-dshm\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.842676 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.842632 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-tmp-dir\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.842887 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.842707 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5250dc2-7ac2-4868-8654-181200be07ce-tls-certs\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.842887 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.842747 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-model-cache\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.943675 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.943640 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-home\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.943914 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.943696 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7t7g\" (UniqueName: \"kubernetes.io/projected/e5250dc2-7ac2-4868-8654-181200be07ce-kube-api-access-h7t7g\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.943914 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.943725 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-dshm\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.943914 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.943745 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-tmp-dir\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.943914 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.943776 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5250dc2-7ac2-4868-8654-181200be07ce-tls-certs\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.943914 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.943801 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-model-cache\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.944182 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.944157 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-home\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.944182 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.944171 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-tmp-dir\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.944276 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.944230 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-model-cache\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.946177 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.946156 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-dshm\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.946445 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.946422 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5250dc2-7ac2-4868-8654-181200be07ce-tls-certs\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:49.952187 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:49.952166 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7t7g\" (UniqueName: \"kubernetes.io/projected/e5250dc2-7ac2-4868-8654-181200be07ce-kube-api-access-h7t7g\") pod \"rate-test-model-35792e-kserve-7f8fd96ccf-4f85f\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:50.063335 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:50.063307 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:39:50.194341 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:50.194235 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f"] Jul 10 18:39:50.197160 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:39:50.197138 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5250dc2_7ac2_4868_8654_181200be07ce.slice/crio-1cd37555fab4cf422493ffa4dd2a3f8632faff9561afec1625a79596892f290e WatchSource:0}: Error finding container 1cd37555fab4cf422493ffa4dd2a3f8632faff9561afec1625a79596892f290e: Status 404 returned error can't find the container with id 1cd37555fab4cf422493ffa4dd2a3f8632faff9561afec1625a79596892f290e Jul 10 18:39:50.542861 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:50.542747 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" event={"ID":"e5250dc2-7ac2-4868-8654-181200be07ce","Type":"ContainerStarted","Data":"54b5daa896cd45d7cc3e4d14d72e7a2df88fb16fcab4a34150bc330486bde3bb"} Jul 10 18:39:50.542861 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:50.542789 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" event={"ID":"e5250dc2-7ac2-4868-8654-181200be07ce","Type":"ContainerStarted","Data":"1cd37555fab4cf422493ffa4dd2a3f8632faff9561afec1625a79596892f290e"} Jul 10 18:39:50.559476 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:50.559425 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" podStartSLOduration=1.559409328 podStartE2EDuration="1.559409328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:50.55913542 +0000 UTC m=+2394.105598562" watchObservedRunningTime="2026-07-10 18:39:50.559409328 +0000 UTC m=+2394.105872517" Jul 10 18:39:55.046549 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:55.046518 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:39:57.060373 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:57.060336 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:39:57.065787 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:57.065763 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:39:57.066201 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:57.066181 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:39:57.072198 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:39:57.072174 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6z78q_159d239e-3d57-4384-95cd-e37f9bdcc401/ovn-acl-logging/0.log" Jul 10 18:40:00.064116 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:40:00.064076 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:40:00.064116 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:40:00.064125 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:40:00.076958 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:40:00.076932 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:40:00.601195 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:40:00.601167 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:40:26.847257 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:40:26.847179 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:40:42.151332 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:40:42.151302 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:40:49.133942 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:40:49.133914 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf"] Jul 10 18:40:50.132232 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:40:50.132204 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f"] Jul 10 18:40:57.840913 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:40:57.840869 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:41:04.154077 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.154023 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" podUID="f38284ab-e4a5-481e-b3b8-de4513360ebd" containerName="main" containerID="cri-o://8d0ecadaacbc71b82391e321ecac410098cc43efd3130fe75168430d002c73d1" gracePeriod=45 Jul 10 18:41:04.413910 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.413825 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:41:04.516333 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.516278 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc89k\" (UniqueName: \"kubernetes.io/projected/f38284ab-e4a5-481e-b3b8-de4513360ebd-kube-api-access-jc89k\") pod \"f38284ab-e4a5-481e-b3b8-de4513360ebd\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " Jul 10 18:41:04.516333 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.516321 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-home\") pod \"f38284ab-e4a5-481e-b3b8-de4513360ebd\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " Jul 10 18:41:04.516634 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.516353 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-dshm\") pod \"f38284ab-e4a5-481e-b3b8-de4513360ebd\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " Jul 10 18:41:04.516634 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.516391 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38284ab-e4a5-481e-b3b8-de4513360ebd-tls-certs\") pod \"f38284ab-e4a5-481e-b3b8-de4513360ebd\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " Jul 10 18:41:04.516634 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.516466 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-model-cache\") pod \"f38284ab-e4a5-481e-b3b8-de4513360ebd\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " Jul 10 18:41:04.516634 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.516482 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-tmp-dir\") pod \"f38284ab-e4a5-481e-b3b8-de4513360ebd\" (UID: \"f38284ab-e4a5-481e-b3b8-de4513360ebd\") " Jul 10 18:41:04.516886 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.516651 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-home" (OuterVolumeSpecName: "home") pod "f38284ab-e4a5-481e-b3b8-de4513360ebd" (UID: "f38284ab-e4a5-481e-b3b8-de4513360ebd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:41:04.516886 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.516826 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-home\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:41:04.516886 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.516801 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-model-cache" (OuterVolumeSpecName: "model-cache") pod "f38284ab-e4a5-481e-b3b8-de4513360ebd" (UID: "f38284ab-e4a5-481e-b3b8-de4513360ebd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:41:04.517056 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.516882 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f38284ab-e4a5-481e-b3b8-de4513360ebd" (UID: "f38284ab-e4a5-481e-b3b8-de4513360ebd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:41:04.518722 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.518689 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-dshm" (OuterVolumeSpecName: "dshm") pod "f38284ab-e4a5-481e-b3b8-de4513360ebd" (UID: "f38284ab-e4a5-481e-b3b8-de4513360ebd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:41:04.518857 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.518740 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38284ab-e4a5-481e-b3b8-de4513360ebd-kube-api-access-jc89k" (OuterVolumeSpecName: "kube-api-access-jc89k") pod "f38284ab-e4a5-481e-b3b8-de4513360ebd" (UID: "f38284ab-e4a5-481e-b3b8-de4513360ebd"). InnerVolumeSpecName "kube-api-access-jc89k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:41:04.519066 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.519036 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38284ab-e4a5-481e-b3b8-de4513360ebd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f38284ab-e4a5-481e-b3b8-de4513360ebd" (UID: "f38284ab-e4a5-481e-b3b8-de4513360ebd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:41:04.618254 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.618170 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-model-cache\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:41:04.618254 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.618245 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-tmp-dir\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:41:04.618254 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.618262 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc89k\" (UniqueName: \"kubernetes.io/projected/f38284ab-e4a5-481e-b3b8-de4513360ebd-kube-api-access-jc89k\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:41:04.618254 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.618274 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38284ab-e4a5-481e-b3b8-de4513360ebd-dshm\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:41:04.618563 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.618287 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38284ab-e4a5-481e-b3b8-de4513360ebd-tls-certs\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:41:04.896487 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.896435 2569 generic.go:356] "Generic (PLEG): container finished" podID="f38284ab-e4a5-481e-b3b8-de4513360ebd" containerID="8d0ecadaacbc71b82391e321ecac410098cc43efd3130fe75168430d002c73d1" exitCode=0 Jul 10 18:41:04.896726 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.896528 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" Jul 10 18:41:04.896726 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.896528 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" event={"ID":"f38284ab-e4a5-481e-b3b8-de4513360ebd","Type":"ContainerDied","Data":"8d0ecadaacbc71b82391e321ecac410098cc43efd3130fe75168430d002c73d1"} Jul 10 18:41:04.896726 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.896604 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf" event={"ID":"f38284ab-e4a5-481e-b3b8-de4513360ebd","Type":"ContainerDied","Data":"1b60af8f75237e91da675d1d6ea2982bb2d23a90db1f886be62dd45922010214"} Jul 10 18:41:04.896726 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.896633 2569 scope.go:117] "RemoveContainer" containerID="8d0ecadaacbc71b82391e321ecac410098cc43efd3130fe75168430d002c73d1" Jul 10 18:41:04.907956 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.907927 2569 scope.go:117] "RemoveContainer" containerID="8d0ecadaacbc71b82391e321ecac410098cc43efd3130fe75168430d002c73d1" Jul 10 18:41:04.908303 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:41:04.908282 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0ecadaacbc71b82391e321ecac410098cc43efd3130fe75168430d002c73d1\": container with ID starting with 8d0ecadaacbc71b82391e321ecac410098cc43efd3130fe75168430d002c73d1 not found: ID does not exist" containerID="8d0ecadaacbc71b82391e321ecac410098cc43efd3130fe75168430d002c73d1" Jul 10 18:41:04.908362 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.908313 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0ecadaacbc71b82391e321ecac410098cc43efd3130fe75168430d002c73d1"} err="failed to get container status \"8d0ecadaacbc71b82391e321ecac410098cc43efd3130fe75168430d002c73d1\": rpc error: code = NotFound desc = could not find container \"8d0ecadaacbc71b82391e321ecac410098cc43efd3130fe75168430d002c73d1\": container with ID starting with 8d0ecadaacbc71b82391e321ecac410098cc43efd3130fe75168430d002c73d1 not found: ID does not exist" Jul 10 18:41:04.920314 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.920262 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf"] Jul 10 18:41:04.923620 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.923581 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-48683a/rate-test-model-48683a-kserve-594f9cb96c-kxxcf"] Jul 10 18:41:04.984002 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:04.983970 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f38284ab-e4a5-481e-b3b8-de4513360ebd" path="/var/lib/kubelet/pods/f38284ab-e4a5-481e-b3b8-de4513360ebd/volumes" Jul 10 18:41:05.151732 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.151583 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" podUID="e5250dc2-7ac2-4868-8654-181200be07ce" containerName="main" containerID="cri-o://54b5daa896cd45d7cc3e4d14d72e7a2df88fb16fcab4a34150bc330486bde3bb" gracePeriod=45 Jul 10 18:41:05.427663 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.427626 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:41:05.527321 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.527271 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-dshm\") pod \"e5250dc2-7ac2-4868-8654-181200be07ce\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " Jul 10 18:41:05.527560 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.527330 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-model-cache\") pod \"e5250dc2-7ac2-4868-8654-181200be07ce\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " Jul 10 18:41:05.527560 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.527384 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-tmp-dir\") pod \"e5250dc2-7ac2-4868-8654-181200be07ce\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " Jul 10 18:41:05.527560 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.527431 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-home\") pod \"e5250dc2-7ac2-4868-8654-181200be07ce\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " Jul 10 18:41:05.527560 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.527451 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5250dc2-7ac2-4868-8654-181200be07ce-tls-certs\") pod \"e5250dc2-7ac2-4868-8654-181200be07ce\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " Jul 10 18:41:05.527560 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.527478 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7t7g\" (UniqueName: \"kubernetes.io/projected/e5250dc2-7ac2-4868-8654-181200be07ce-kube-api-access-h7t7g\") pod \"e5250dc2-7ac2-4868-8654-181200be07ce\" (UID: \"e5250dc2-7ac2-4868-8654-181200be07ce\") " Jul 10 18:41:05.527977 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.527655 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5250dc2-7ac2-4868-8654-181200be07ce" (UID: "e5250dc2-7ac2-4868-8654-181200be07ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:41:05.527977 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.527689 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5250dc2-7ac2-4868-8654-181200be07ce" (UID: "e5250dc2-7ac2-4868-8654-181200be07ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:41:05.527977 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.527713 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-home" (OuterVolumeSpecName: "home") pod "e5250dc2-7ac2-4868-8654-181200be07ce" (UID: "e5250dc2-7ac2-4868-8654-181200be07ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:41:05.527977 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.527801 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-model-cache\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:41:05.527977 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.527853 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-tmp-dir\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:41:05.530016 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.529974 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5250dc2-7ac2-4868-8654-181200be07ce-kube-api-access-h7t7g" (OuterVolumeSpecName: "kube-api-access-h7t7g") pod "e5250dc2-7ac2-4868-8654-181200be07ce" (UID: "e5250dc2-7ac2-4868-8654-181200be07ce"). InnerVolumeSpecName "kube-api-access-h7t7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:41:05.530016 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.530008 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-dshm" (OuterVolumeSpecName: "dshm") pod "e5250dc2-7ac2-4868-8654-181200be07ce" (UID: "e5250dc2-7ac2-4868-8654-181200be07ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:41:05.530016 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.529965 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5250dc2-7ac2-4868-8654-181200be07ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5250dc2-7ac2-4868-8654-181200be07ce" (UID: "e5250dc2-7ac2-4868-8654-181200be07ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:41:05.629189 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.629136 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-home\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:41:05.629189 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.629175 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5250dc2-7ac2-4868-8654-181200be07ce-tls-certs\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:41:05.629189 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.629189 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7t7g\" (UniqueName: \"kubernetes.io/projected/e5250dc2-7ac2-4868-8654-181200be07ce-kube-api-access-h7t7g\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:41:05.629189 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.629199 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5250dc2-7ac2-4868-8654-181200be07ce-dshm\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:41:05.904669 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.904623 2569 generic.go:356] "Generic (PLEG): container finished" podID="e5250dc2-7ac2-4868-8654-181200be07ce" containerID="54b5daa896cd45d7cc3e4d14d72e7a2df88fb16fcab4a34150bc330486bde3bb" exitCode=0 Jul 10 18:41:05.904669 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.904674 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" Jul 10 18:41:05.905021 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.904710 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" event={"ID":"e5250dc2-7ac2-4868-8654-181200be07ce","Type":"ContainerDied","Data":"54b5daa896cd45d7cc3e4d14d72e7a2df88fb16fcab4a34150bc330486bde3bb"} Jul 10 18:41:05.905021 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.904762 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f" event={"ID":"e5250dc2-7ac2-4868-8654-181200be07ce","Type":"ContainerDied","Data":"1cd37555fab4cf422493ffa4dd2a3f8632faff9561afec1625a79596892f290e"} Jul 10 18:41:05.905021 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.904785 2569 scope.go:117] "RemoveContainer" containerID="54b5daa896cd45d7cc3e4d14d72e7a2df88fb16fcab4a34150bc330486bde3bb" Jul 10 18:41:05.915800 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.915766 2569 scope.go:117] "RemoveContainer" containerID="54b5daa896cd45d7cc3e4d14d72e7a2df88fb16fcab4a34150bc330486bde3bb" Jul 10 18:41:05.916216 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:41:05.916193 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b5daa896cd45d7cc3e4d14d72e7a2df88fb16fcab4a34150bc330486bde3bb\": container with ID starting with 54b5daa896cd45d7cc3e4d14d72e7a2df88fb16fcab4a34150bc330486bde3bb not found: ID does not exist" containerID="54b5daa896cd45d7cc3e4d14d72e7a2df88fb16fcab4a34150bc330486bde3bb" Jul 10 18:41:05.916275 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.916228 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b5daa896cd45d7cc3e4d14d72e7a2df88fb16fcab4a34150bc330486bde3bb"} err="failed to get container status \"54b5daa896cd45d7cc3e4d14d72e7a2df88fb16fcab4a34150bc330486bde3bb\": rpc error: code = NotFound desc = could not find container \"54b5daa896cd45d7cc3e4d14d72e7a2df88fb16fcab4a34150bc330486bde3bb\": container with ID starting with 54b5daa896cd45d7cc3e4d14d72e7a2df88fb16fcab4a34150bc330486bde3bb not found: ID does not exist" Jul 10 18:41:05.928309 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.928260 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f"] Jul 10 18:41:05.931898 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:05.931845 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-35792e/rate-test-model-35792e-kserve-7f8fd96ccf-4f85f"] Jul 10 18:41:06.282608 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.282512 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7151-140327022566256"] Jul 10 18:41:06.282920 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.282906 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5250dc2-7ac2-4868-8654-181200be07ce" containerName="main" Jul 10 18:41:06.282920 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.282921 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5250dc2-7ac2-4868-8654-181200be07ce" containerName="main" Jul 10 18:41:06.283017 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.282944 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38284ab-e4a5-481e-b3b8-de4513360ebd" containerName="main" Jul 10 18:41:06.283017 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.282949 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38284ab-e4a5-481e-b3b8-de4513360ebd" containerName="main" Jul 10 18:41:06.283017 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.283012 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f38284ab-e4a5-481e-b3b8-de4513360ebd" containerName="main" Jul 10 18:41:06.283117 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.283025 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5250dc2-7ac2-4868-8654-181200be07ce" containerName="main" Jul 10 18:41:06.287619 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.287587 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-140327022566256" Jul 10 18:41:06.290474 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.290437 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-xk98h\"" Jul 10 18:41:06.291677 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.291643 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7151-140327022566256"] Jul 10 18:41:06.336222 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.336180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8kmn\" (UniqueName: \"kubernetes.io/projected/fb3d4fc8-2ffd-441e-9af1-01c85cc231e1-kube-api-access-c8kmn\") pod \"test-curl-7151-140327022566256\" (UID: \"fb3d4fc8-2ffd-441e-9af1-01c85cc231e1\") " pod="openshift-ingress/test-curl-7151-140327022566256" Jul 10 18:41:06.437044 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.437000 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8kmn\" (UniqueName: \"kubernetes.io/projected/fb3d4fc8-2ffd-441e-9af1-01c85cc231e1-kube-api-access-c8kmn\") pod \"test-curl-7151-140327022566256\" (UID: \"fb3d4fc8-2ffd-441e-9af1-01c85cc231e1\") " pod="openshift-ingress/test-curl-7151-140327022566256" Jul 10 18:41:06.446175 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.446128 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8kmn\" (UniqueName: \"kubernetes.io/projected/fb3d4fc8-2ffd-441e-9af1-01c85cc231e1-kube-api-access-c8kmn\") pod \"test-curl-7151-140327022566256\" (UID: \"fb3d4fc8-2ffd-441e-9af1-01c85cc231e1\") " pod="openshift-ingress/test-curl-7151-140327022566256" Jul 10 18:41:06.600766 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.600718 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-140327022566256" Jul 10 18:41:06.743802 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.743778 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7151-140327022566256"] Jul 10 18:41:06.746391 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:41:06.746362 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb3d4fc8_2ffd_441e_9af1_01c85cc231e1.slice/crio-6a8f49faf07c3c16b9ed2ea00a55e302fc6c8ccb746bff55695aa6c9c8dd41b2 WatchSource:0}: Error finding container 6a8f49faf07c3c16b9ed2ea00a55e302fc6c8ccb746bff55695aa6c9c8dd41b2: Status 404 returned error can't find the container with id 6a8f49faf07c3c16b9ed2ea00a55e302fc6c8ccb746bff55695aa6c9c8dd41b2 Jul 10 18:41:06.747673 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.747653 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:41:06.912538 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.912416 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7151-140327022566256" event={"ID":"fb3d4fc8-2ffd-441e-9af1-01c85cc231e1","Type":"ContainerStarted","Data":"6a8f49faf07c3c16b9ed2ea00a55e302fc6c8ccb746bff55695aa6c9c8dd41b2"} Jul 10 18:41:06.985664 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:06.985610 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5250dc2-7ac2-4868-8654-181200be07ce" path="/var/lib/kubelet/pods/e5250dc2-7ac2-4868-8654-181200be07ce/volumes" Jul 10 18:41:07.919963 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:07.919921 2569 generic.go:356] "Generic (PLEG): container finished" podID="fb3d4fc8-2ffd-441e-9af1-01c85cc231e1" containerID="924c74260b331fe5b8e6648d8f9f69215d0ff38633e58525e5826d7b62b0a4ad" exitCode=0 Jul 10 18:41:07.919963 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:07.919973 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7151-140327022566256" event={"ID":"fb3d4fc8-2ffd-441e-9af1-01c85cc231e1","Type":"ContainerDied","Data":"924c74260b331fe5b8e6648d8f9f69215d0ff38633e58525e5826d7b62b0a4ad"} Jul 10 18:41:09.077961 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:09.077919 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-140327022566256" Jul 10 18:41:09.122205 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:09.122176 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7151-140327022566256_fb3d4fc8-2ffd-441e-9af1-01c85cc231e1/test-curl-7151-140327022566256/0.log" Jul 10 18:41:09.149584 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:09.149544 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7151-140327022566256"] Jul 10 18:41:09.153949 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:09.153911 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7151-140327022566256"] Jul 10 18:41:09.162162 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:09.162124 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8kmn\" (UniqueName: \"kubernetes.io/projected/fb3d4fc8-2ffd-441e-9af1-01c85cc231e1-kube-api-access-c8kmn\") pod \"fb3d4fc8-2ffd-441e-9af1-01c85cc231e1\" (UID: \"fb3d4fc8-2ffd-441e-9af1-01c85cc231e1\") " Jul 10 18:41:09.164599 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:09.164559 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3d4fc8-2ffd-441e-9af1-01c85cc231e1-kube-api-access-c8kmn" (OuterVolumeSpecName: "kube-api-access-c8kmn") pod "fb3d4fc8-2ffd-441e-9af1-01c85cc231e1" (UID: "fb3d4fc8-2ffd-441e-9af1-01c85cc231e1"). InnerVolumeSpecName "kube-api-access-c8kmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:41:09.263300 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:09.263207 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8kmn\" (UniqueName: \"kubernetes.io/projected/fb3d4fc8-2ffd-441e-9af1-01c85cc231e1-kube-api-access-c8kmn\") on node \"ip-10-0-136-200.ec2.internal\" DevicePath \"\"" Jul 10 18:41:09.931909 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:09.931871 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-140327022566256" Jul 10 18:41:09.931909 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:09.931893 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a8f49faf07c3c16b9ed2ea00a55e302fc6c8ccb746bff55695aa6c9c8dd41b2" Jul 10 18:41:09.942910 ip-10-0-136-200 kubenswrapper[2569]: E0710 18:41:09.942870 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7151-140327022566256\" is forbidden: User \"system:node:ip-10-0-136-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-200.ec2.internal' and this object" podUID="fb3d4fc8-2ffd-441e-9af1-01c85cc231e1" pod="openshift-ingress/test-curl-7151-140327022566256" Jul 10 18:41:10.984559 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:10.984516 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb3d4fc8-2ffd-441e-9af1-01c85cc231e1" path="/var/lib/kubelet/pods/fb3d4fc8-2ffd-441e-9af1-01c85cc231e1/volumes" Jul 10 18:41:13.647378 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:13.647340 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:41:27.444456 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:27.444412 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tlqnq"] Jul 10 18:41:29.625882 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:29.625848 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb6dcc87f-l8rcq_d6540ad8-78ea-48aa-8d3c-08413c2b7cc4/copy-manifests/0.log" Jul 10 18:41:29.633236 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:29.633197 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb6dcc87f-l8rcq_d6540ad8-78ea-48aa-8d3c-08413c2b7cc4/manager/0.log" Jul 10 18:41:31.652412 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:31.652378 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qj5mh_cb142d01-2268-4d06-85a4-a01c17882f4f/kuadrant-console-plugin/0.log" Jul 10 18:41:32.024063 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:32.023974 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tlqnq_d8237581-fd48-4441-8145-a1035356193e/limitador/0.log" Jul 10 18:41:32.802144 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:32.802114 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5486d894ff-mrqjk_9de0c53a-b50e-4117-9099-0356c268cde5/kube-auth-proxy/0.log" Jul 10 18:41:33.260088 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:33.260000 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b875d77bd-5c2fl_14d16fa6-3622-410f-acac-d4ddfe4704b2/router/0.log" Jul 10 18:41:33.820162 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:33.820125 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4fpnw_27ce69f0-5289-4897-99c2-5f336c41b810/main/0.log" Jul 10 18:41:34.171802 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:34.171674 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-zmkq8_a4438805-1b27-49c7-a434-b6ae614fb73d/main/0.log" Jul 10 18:41:38.098059 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.098001 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5tczq/must-gather-r2rj7"] Jul 10 18:41:38.098575 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.098448 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb3d4fc8-2ffd-441e-9af1-01c85cc231e1" containerName="test-curl-7151-140327022566256" Jul 10 18:41:38.098575 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.098464 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb3d4fc8-2ffd-441e-9af1-01c85cc231e1" containerName="test-curl-7151-140327022566256" Jul 10 18:41:38.098575 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.098544 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb3d4fc8-2ffd-441e-9af1-01c85cc231e1" containerName="test-curl-7151-140327022566256" Jul 10 18:41:38.101753 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.101722 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tczq/must-gather-r2rj7" Jul 10 18:41:38.104434 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.104393 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5tczq\"/\"openshift-service-ca.crt\"" Jul 10 18:41:38.104585 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.104451 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5tczq\"/\"kube-root-ca.crt\"" Jul 10 18:41:38.105385 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.105360 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5tczq\"/\"default-dockercfg-rt4qw\"" Jul 10 18:41:38.110569 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.110530 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tczq/must-gather-r2rj7"] Jul 10 18:41:38.153896 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.153846 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr9qx\" (UniqueName: \"kubernetes.io/projected/5231c995-b3fd-4fb3-b671-4e036586bd73-kube-api-access-zr9qx\") pod \"must-gather-r2rj7\" (UID: \"5231c995-b3fd-4fb3-b671-4e036586bd73\") " pod="openshift-must-gather-5tczq/must-gather-r2rj7" Jul 10 18:41:38.154092 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.153978 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5231c995-b3fd-4fb3-b671-4e036586bd73-must-gather-output\") pod \"must-gather-r2rj7\" (UID: \"5231c995-b3fd-4fb3-b671-4e036586bd73\") " pod="openshift-must-gather-5tczq/must-gather-r2rj7" Jul 10 18:41:38.255059 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.255010 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr9qx\" (UniqueName: \"kubernetes.io/projected/5231c995-b3fd-4fb3-b671-4e036586bd73-kube-api-access-zr9qx\") pod \"must-gather-r2rj7\" (UID: \"5231c995-b3fd-4fb3-b671-4e036586bd73\") " pod="openshift-must-gather-5tczq/must-gather-r2rj7" Jul 10 18:41:38.255255 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.255097 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5231c995-b3fd-4fb3-b671-4e036586bd73-must-gather-output\") pod \"must-gather-r2rj7\" (UID: \"5231c995-b3fd-4fb3-b671-4e036586bd73\") " pod="openshift-must-gather-5tczq/must-gather-r2rj7" Jul 10 18:41:38.255414 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.255398 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5231c995-b3fd-4fb3-b671-4e036586bd73-must-gather-output\") pod \"must-gather-r2rj7\" (UID: \"5231c995-b3fd-4fb3-b671-4e036586bd73\") " pod="openshift-must-gather-5tczq/must-gather-r2rj7" Jul 10 18:41:38.263469 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.263430 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr9qx\" (UniqueName: \"kubernetes.io/projected/5231c995-b3fd-4fb3-b671-4e036586bd73-kube-api-access-zr9qx\") pod \"must-gather-r2rj7\" (UID: \"5231c995-b3fd-4fb3-b671-4e036586bd73\") " pod="openshift-must-gather-5tczq/must-gather-r2rj7" Jul 10 18:41:38.414723 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.414626 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tczq/must-gather-r2rj7" Jul 10 18:41:38.571334 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:38.571293 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tczq/must-gather-r2rj7"] Jul 10 18:41:38.576003 ip-10-0-136-200 kubenswrapper[2569]: W0710 18:41:38.575954 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5231c995_b3fd_4fb3_b671_4e036586bd73.slice/crio-446926983d1e8143a14db56041b95ab0af457ecb8e0a560590ebda34dd27ab29 WatchSource:0}: Error finding container 446926983d1e8143a14db56041b95ab0af457ecb8e0a560590ebda34dd27ab29: Status 404 returned error can't find the container with id 446926983d1e8143a14db56041b95ab0af457ecb8e0a560590ebda34dd27ab29 Jul 10 18:41:39.082362 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:39.082322 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tczq/must-gather-r2rj7" event={"ID":"5231c995-b3fd-4fb3-b671-4e036586bd73","Type":"ContainerStarted","Data":"446926983d1e8143a14db56041b95ab0af457ecb8e0a560590ebda34dd27ab29"} Jul 10 18:41:40.094635 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:40.094588 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tczq/must-gather-r2rj7" event={"ID":"5231c995-b3fd-4fb3-b671-4e036586bd73","Type":"ContainerStarted","Data":"480124d0ac914f840bfe56b50e07bdc43f3aeb5ff9d0585fe358714f10042af1"} Jul 10 18:41:40.094635 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:40.094647 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tczq/must-gather-r2rj7" event={"ID":"5231c995-b3fd-4fb3-b671-4e036586bd73","Type":"ContainerStarted","Data":"8408fc25633ffd675dcc5d375c4989b7b02c41c4283871671a7c62e30ef69834"} Jul 10 18:41:40.114894 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:40.114783 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5tczq/must-gather-r2rj7" podStartSLOduration=1.04542397 podStartE2EDuration="2.114747184s" totalImagesPullingTime="1.069323214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:40.111200116 +0000 UTC m=+2503.657663292" watchObservedRunningTime="2026-07-10 18:41:40.114747184 +0000 UTC m=+2503.661210329" Jul 10 18:41:41.403234 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:41.403196 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g8xxt_12a0c5b8-e1de-44de-a231-5a56fa8a1d23/global-pull-secret-syncer/0.log" Jul 10 18:41:41.545117 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:41.545077 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lbc2g_fb493400-515b-4ed8-934f-038f1aae45ed/konnectivity-agent/0.log" Jul 10 18:41:41.634655 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:41.634613 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-200.ec2.internal_4f403d9863d495b5a87ed67af63e35db/haproxy/0.log" Jul 10 18:41:47.132704 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:47.132654 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qj5mh_cb142d01-2268-4d06-85a4-a01c17882f4f/kuadrant-console-plugin/0.log" Jul 10 18:41:47.383941 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:47.383349 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tlqnq_d8237581-fd48-4441-8145-a1035356193e/limitador/0.log" Jul 10 18:41:49.602405 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:49.602304 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-x4vcg_5a25093f-9aee-4ca0-8996-08949e85bcf7/kube-state-metrics/0.log" Jul 10 18:41:49.638855 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:49.638800 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-x4vcg_5a25093f-9aee-4ca0-8996-08949e85bcf7/kube-rbac-proxy-main/0.log" Jul 10 18:41:49.667151 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:49.667109 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-x4vcg_5a25093f-9aee-4ca0-8996-08949e85bcf7/kube-rbac-proxy-self/0.log" Jul 10 18:41:49.704445 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:49.704415 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8f8458798-2klml_5680deaa-f23f-49c9-8aee-8995859cac4a/metrics-server/0.log" Jul 10 18:41:49.767844 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:49.767774 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8lg4v_c91b6298-ea96-4042-8fc8-7145c6c87aa6/node-exporter/0.log" Jul 10 18:41:49.793410 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:49.793379 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8lg4v_c91b6298-ea96-4042-8fc8-7145c6c87aa6/kube-rbac-proxy/0.log" Jul 10 18:41:49.824349 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:49.824314 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8lg4v_c91b6298-ea96-4042-8fc8-7145c6c87aa6/init-textfile/0.log" Jul 10 18:41:50.173301 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:50.173241 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d23fdb3-3a8e-4274-a6d4-1145264d2167/prometheus/0.log" Jul 10 18:41:50.205871 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:50.205793 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d23fdb3-3a8e-4274-a6d4-1145264d2167/config-reloader/0.log" Jul 10 18:41:50.245359 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:50.245323 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d23fdb3-3a8e-4274-a6d4-1145264d2167/thanos-sidecar/0.log" Jul 10 18:41:50.321876 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:50.321841 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d23fdb3-3a8e-4274-a6d4-1145264d2167/kube-rbac-proxy-web/0.log" Jul 10 18:41:50.391012 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:50.390914 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d23fdb3-3a8e-4274-a6d4-1145264d2167/kube-rbac-proxy/0.log" Jul 10 18:41:50.422546 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:50.422501 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d23fdb3-3a8e-4274-a6d4-1145264d2167/kube-rbac-proxy-thanos/0.log" Jul 10 18:41:50.450109 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:50.450082 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d23fdb3-3a8e-4274-a6d4-1145264d2167/init-config-reloader/0.log" Jul 10 18:41:50.481129 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:50.481091 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-kgnvb_91c7ad09-d763-4204-8771-8324e0b5d0cb/prometheus-operator/0.log" Jul 10 18:41:50.500309 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:50.500276 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-kgnvb_91c7ad09-d763-4204-8771-8324e0b5d0cb/kube-rbac-proxy/0.log" Jul 10 18:41:50.527950 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:50.527916 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-bvh74_2eb92c4e-4ea0-427d-82bd-ccdc82d2b2b9/prometheus-operator-admission-webhook/0.log" Jul 10 18:41:52.388645 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.388603 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh"] Jul 10 18:41:52.395035 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.394996 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.401060 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.401015 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh"] Jul 10 18:41:52.529343 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.529304 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/686a571a-b1f4-4e22-9119-f904ee798e3c-proc\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.529554 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.529427 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/686a571a-b1f4-4e22-9119-f904ee798e3c-lib-modules\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.529554 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.529535 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/686a571a-b1f4-4e22-9119-f904ee798e3c-podres\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.529680 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.529569 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvn9f\" (UniqueName: \"kubernetes.io/projected/686a571a-b1f4-4e22-9119-f904ee798e3c-kube-api-access-gvn9f\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.529680 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.529596 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/686a571a-b1f4-4e22-9119-f904ee798e3c-sys\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.630507 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.630454 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/686a571a-b1f4-4e22-9119-f904ee798e3c-proc\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.630845 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.630682 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/686a571a-b1f4-4e22-9119-f904ee798e3c-proc\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.631024 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.630947 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/686a571a-b1f4-4e22-9119-f904ee798e3c-lib-modules\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.631228 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.631208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/686a571a-b1f4-4e22-9119-f904ee798e3c-podres\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.631352 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.631337 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn9f\" (UniqueName: \"kubernetes.io/projected/686a571a-b1f4-4e22-9119-f904ee798e3c-kube-api-access-gvn9f\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.631447 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.631422 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/686a571a-b1f4-4e22-9119-f904ee798e3c-podres\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.631510 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.631104 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/686a571a-b1f4-4e22-9119-f904ee798e3c-lib-modules\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.631601 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.631578 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/686a571a-b1f4-4e22-9119-f904ee798e3c-sys\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.631910 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.631889 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/686a571a-b1f4-4e22-9119-f904ee798e3c-sys\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.640896 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.640771 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvn9f\" (UniqueName: \"kubernetes.io/projected/686a571a-b1f4-4e22-9119-f904ee798e3c-kube-api-access-gvn9f\") pod \"perf-node-gather-daemonset-thvfh\" (UID: \"686a571a-b1f4-4e22-9119-f904ee798e3c\") " pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.715433 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.715394 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:52.731041 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.730743 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/2.log" Jul 10 18:41:52.737171 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.737136 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x72w7_a4f78d82-214c-4612-adc5-32497c685a36/console-operator/3.log" Jul 10 18:41:52.925306 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:52.924122 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh"] Jul 10 18:41:53.196426 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:53.196328 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" event={"ID":"686a571a-b1f4-4e22-9119-f904ee798e3c","Type":"ContainerStarted","Data":"34f318262454ca3c8e1bee69b5be126e1d19b77ceb46a1e8f1724f0dd3463079"} Jul 10 18:41:53.196426 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:53.196417 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" event={"ID":"686a571a-b1f4-4e22-9119-f904ee798e3c","Type":"ContainerStarted","Data":"09d62205232d14fca85ead5bbe75f87ec4878d2679b605d69274fca707d9afe1"} Jul 10 18:41:53.196688 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:53.196544 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:41:53.215951 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:53.215864 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" podStartSLOduration=1.215802842 podStartE2EDuration="1.215802842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:53.212435668 +0000 UTC m=+2516.758898813" watchObservedRunningTime="2026-07-10 18:41:53.215802842 +0000 UTC m=+2516.762265984" Jul 10 18:41:53.993436 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:53.993403 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-gzbnj_c81d7714-2c32-4459-9221-41271562f618/volume-data-source-validator/0.log" Jul 10 18:41:55.203189 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:55.203147 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t6b2r_b6626849-dd7f-41ff-b019-04d42ae9229d/dns/0.log" Jul 10 18:41:55.225840 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:55.225796 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t6b2r_b6626849-dd7f-41ff-b019-04d42ae9229d/kube-rbac-proxy/0.log" Jul 10 18:41:55.278712 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:55.278685 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g7nkb_456cd2fa-e3a0-40f3-b865-15d505e82bbe/dns-node-resolver/0.log" Jul 10 18:41:55.898145 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:55.898096 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5fb7478855-8rd8t_94439f31-71d0-49e6-81b7-27f670e8ef16/registry/0.log" Jul 10 18:41:55.952064 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:55.952008 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f4xrb_e7d64404-7c29-458a-bc00-2a745b9cbc85/node-ca/0.log" Jul 10 18:41:57.213079 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:57.213052 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5486d894ff-mrqjk_9de0c53a-b50e-4117-9099-0356c268cde5/kube-auth-proxy/0.log" Jul 10 18:41:57.397912 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:57.397876 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b875d77bd-5c2fl_14d16fa6-3622-410f-acac-d4ddfe4704b2/router/0.log" Jul 10 18:41:58.182429 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:58.182385 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m7jqp_c68aaf65-4c9a-4016-9200-088deed583b9/serve-healthcheck-canary/0.log" Jul 10 18:41:58.747273 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:58.747229 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-lt7vz_3404e0d1-d53d-4ccf-8b88-36893095d733/insights-operator/0.log" Jul 10 18:41:58.753523 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:58.748450 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-lt7vz_3404e0d1-d53d-4ccf-8b88-36893095d733/insights-operator/1.log" Jul 10 18:41:58.854188 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:58.854142 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d86p4_ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca/kube-rbac-proxy/0.log" Jul 10 18:41:58.877081 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:58.877048 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d86p4_ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca/exporter/0.log" Jul 10 18:41:58.900087 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:58.900055 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d86p4_ea76ed28-d4d8-49ef-b8b4-4c5a7f9c38ca/extractor/0.log" Jul 10 18:41:59.217648 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:41:59.217604 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5tczq/perf-node-gather-daemonset-thvfh" Jul 10 18:42:01.998348 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:42:01.998223 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb6dcc87f-l8rcq_d6540ad8-78ea-48aa-8d3c-08413c2b7cc4/manager/0.log" Jul 10 18:42:02.020220 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:42:02.020187 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb6dcc87f-l8rcq_d6540ad8-78ea-48aa-8d3c-08413c2b7cc4/copy-manifests/0.log" Jul 10 18:42:03.306763 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:42:03.306725 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d79ff4f99-xhm9m_4d8e150f-f183-4535-8250-d37af204e1c8/manager/0.log" Jul 10 18:42:04.509617 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:42:04.509574 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5486d894ff-mrqjk_9de0c53a-b50e-4117-9099-0356c268cde5/kube-auth-proxy/0.log" Jul 10 18:42:04.651925 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:42:04.651888 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b875d77bd-5c2fl_14d16fa6-3622-410f-acac-d4ddfe4704b2/router/0.log" Jul 10 18:42:06.206457 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:42:06.206424 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5486d894ff-mrqjk_9de0c53a-b50e-4117-9099-0356c268cde5/kube-auth-proxy/0.log" Jul 10 18:42:06.349174 ip-10-0-136-200 kubenswrapper[2569]: I0710 18:42:06.349141 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b875d77bd-5c2fl_14d16fa6-3622-410f-acac-d4ddfe4704b2/router/0.log"