Jul 10 17:36:54.387446 ip-10-0-137-103 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 17:36:54.387460 ip-10-0-137-103 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 17:36:54.387471 ip-10-0-137-103 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 17:36:54.387825 ip-10-0-137-103 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 17:37:04.472581 ip-10-0-137-103 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 17:37:04.472600 ip-10-0-137-103 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c4aab96e00b0495d81e82a98fd791656 -- Jul 10 17:39:09.275565 ip-10-0-137-103 systemd[1]: Starting Kubernetes Kubelet... Jul 10 17:39:09.676745 ip-10-0-137-103 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:39:09.676745 ip-10-0-137-103 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 17:39:09.676745 ip-10-0-137-103 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:39:09.676745 ip-10-0-137-103 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 17:39:09.676745 ip-10-0-137-103 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:39:09.677475 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.676812 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 17:39:09.682336 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682320 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:39:09.682336 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682336 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682340 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682344 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682348 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682353 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682356 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682359 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682362 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682365 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682368 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682371 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682373 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682376 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682378 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682381 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682384 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682386 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682389 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682391 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:39:09.682415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682394 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682396 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682399 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682402 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682404 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682407 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682409 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682412 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682415 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682418 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682420 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682424 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682427 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682429 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682432 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682435 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682437 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682440 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682442 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682445 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:39:09.682899 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682447 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682450 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682452 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682454 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682457 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682460 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682463 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682465 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682468 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682470 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682473 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682476 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682479 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682481 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682484 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682486 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682489 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682491 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682494 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:39:09.683420 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682496 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682500 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682504 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682507 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682510 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682512 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682515 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682518 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682521 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682523 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682526 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682528 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682531 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682534 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682537 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682539 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682541 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682544 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682546 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:39:09.683879 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682549 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682551 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682554 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682556 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682558 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682562 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682565 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682568 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682571 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682573 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682576 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682579 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682581 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682584 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682588 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682592 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682595 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682598 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682601 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:39:09.684339 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682603 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.682606 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683856 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683863 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683866 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683869 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683873 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683877 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683880 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683883 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683886 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683889 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683891 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683894 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683897 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683899 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683902 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683905 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683907 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683910 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:39:09.684797 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683913 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683915 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683918 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683921 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683923 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683926 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683928 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683931 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683934 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683936 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683939 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683941 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683944 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683946 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683949 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683951 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683954 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683956 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683959 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683961 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:39:09.685310 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683970 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683973 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683975 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683978 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683980 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683983 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683986 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683988 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683991 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683993 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683996 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.683998 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684002 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684004 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684007 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684010 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684012 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684015 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684018 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:39:09.685794 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684021 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684025 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684028 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684031 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684033 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684036 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684038 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684041 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684044 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684046 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684049 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684052 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684054 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684057 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684060 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684062 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684067 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684070 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684073 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684076 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:39:09.686249 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684079 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684081 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684084 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684087 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684089 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684092 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684097 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684100 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684103 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684106 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684109 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684112 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684115 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684117 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684120 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684123 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684125 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684128 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684131 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:39:09.686771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684134 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684136 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684139 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684567 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684573 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684576 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684578 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684582 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684584 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684588 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684590 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684593 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684596 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684599 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684601 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684604 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684606 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684608 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684612 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684614 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:39:09.687238 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684617 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684620 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684622 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684625 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684627 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684630 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684633 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684635 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684638 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684640 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684644 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684647 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684649 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684651 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684654 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684657 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684659 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684661 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684664 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684666 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:39:09.687765 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684669 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684672 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684674 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684676 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684679 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684681 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684685 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684691 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684693 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684695 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684698 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684701 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684704 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684706 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684708 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684711 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684713 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684716 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684718 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684721 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:39:09.688241 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684725 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684728 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684731 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684733 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684736 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684739 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684741 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684744 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684746 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684748 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684751 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684753 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684756 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684759 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684761 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684764 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684766 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684770 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684772 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684776 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:39:09.688734 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684778 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684781 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684784 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684787 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684790 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684792 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684795 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684797 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684799 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684803 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684807 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684810 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684813 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684816 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684818 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684822 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684825 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684828 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684831 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:39:09.689236 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684834 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684837 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.684839 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684920 2556 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684931 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684939 2556 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684946 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684964 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684969 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684974 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684978 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684981 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684985 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684989 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684992 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684995 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.684999 2556 flags.go:64] FLAG: --cgroup-root="" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685002 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685004 2556 flags.go:64] FLAG: --client-ca-file="" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685007 2556 flags.go:64] FLAG: --cloud-provider="external" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685010 2556 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685015 2556 flags.go:64] FLAG: --cluster-domain="" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685018 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685021 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 17:39:09.689723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685023 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685027 2556 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685031 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685033 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685036 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685039 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685042 2556 flags.go:64] FLAG: --contention-profiling="false" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685045 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685048 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685051 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685054 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685058 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685061 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685064 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685067 2556 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685069 2556 flags.go:64] FLAG: --enable-server="true" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685073 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685077 2556 flags.go:64] FLAG: --event-burst="100" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685080 2556 flags.go:64] FLAG: --event-qps="50" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685083 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685087 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685090 2556 flags.go:64] FLAG: --eviction-hard="" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685094 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685097 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685100 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 17:39:09.690313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685103 2556 flags.go:64] FLAG: --eviction-soft="" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685105 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685108 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685111 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685114 2556 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685117 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685120 2556 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685123 2556 flags.go:64] FLAG: --feature-gates="" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685126 2556 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685129 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685132 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685135 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685138 2556 flags.go:64] FLAG: --healthz-port="10248" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685141 2556 flags.go:64] FLAG: --help="false" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685144 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-137-103.ec2.internal" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685147 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685150 2556 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685153 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685156 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685159 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685162 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685164 2556 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685167 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 17:39:09.690932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685170 2556 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685173 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685176 2556 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685179 2556 flags.go:64] FLAG: --kube-reserved="" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685184 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685187 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685190 2556 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685193 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685195 2556 flags.go:64] FLAG: --lock-file="" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685198 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685201 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685203 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685209 2556 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685211 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685214 2556 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685217 2556 flags.go:64] FLAG: --logging-format="text" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685219 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685223 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685225 2556 flags.go:64] FLAG: --manifest-url="" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685228 2556 flags.go:64] FLAG: --manifest-url-header="" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685232 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685235 2556 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685239 2556 flags.go:64] FLAG: --max-pods="110" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685242 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685253 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 17:39:09.691617 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685256 2556 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685275 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685281 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685284 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685287 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685294 2556 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685297 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685300 2556 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685303 2556 flags.go:64] FLAG: --pod-cidr="" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685307 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685313 2556 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685317 2556 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685321 2556 flags.go:64] FLAG: --pods-per-core="0" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685323 2556 flags.go:64] FLAG: --port="10250" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685328 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685331 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-072209d07c7e3cb78" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685334 2556 flags.go:64] FLAG: --qos-reserved="" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685337 2556 flags.go:64] FLAG: --read-only-port="10255" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685340 2556 flags.go:64] FLAG: --register-node="true" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685343 2556 flags.go:64] FLAG: --register-with-taints="" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685346 2556 flags.go:64] FLAG: --registry-burst="10" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685349 2556 flags.go:64] FLAG: --registry-qps="5" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685352 2556 flags.go:64] FLAG: --reserved-cpus="" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685355 2556 flags.go:64] FLAG: --reserved-memory="" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685358 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 17:39:09.692327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685361 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685364 2556 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685367 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685369 2556 flags.go:64] FLAG: --runonce="false" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685372 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685375 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685378 2556 flags.go:64] FLAG: --seccomp-default="false" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685380 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685383 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685386 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685389 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685393 2556 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685395 2556 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685398 2556 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685401 2556 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685403 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685406 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685410 2556 flags.go:64] FLAG: --system-cgroups="" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685413 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685419 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685422 2556 flags.go:64] FLAG: --tls-cert-file="" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685426 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685430 2556 flags.go:64] FLAG: --tls-min-version="" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685433 2556 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685435 2556 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 17:39:09.693081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685438 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685441 2556 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685444 2556 flags.go:64] FLAG: --v="2" Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685448 2556 flags.go:64] FLAG: --version="false" Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685451 2556 flags.go:64] FLAG: --vmodule="" Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685455 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.685458 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685596 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685600 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685603 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685606 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685609 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685612 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685615 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685617 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685620 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685622 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685625 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685628 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685630 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685633 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685635 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:39:09.693801 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685638 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685640 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685643 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685646 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685648 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685651 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685655 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685657 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685660 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685662 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685665 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685669 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685672 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685675 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685678 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685681 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685684 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685687 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685689 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:39:09.694426 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685692 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685694 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685697 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685700 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685702 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685705 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685707 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685709 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685712 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685714 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685716 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685719 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685721 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685724 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685728 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685731 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685734 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685737 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685739 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685743 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:39:09.694964 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685746 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685748 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685751 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685753 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685756 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685759 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685761 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685764 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685766 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685769 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685772 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685774 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685777 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685779 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685782 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685784 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685786 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685789 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685792 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:39:09.695527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685794 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685797 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685799 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685802 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685805 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685808 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685810 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685812 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685815 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685818 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685821 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685824 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685828 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685830 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685833 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685837 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685840 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685843 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685846 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685848 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:39:09.696066 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685851 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685853 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685856 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685860 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685862 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.685865 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.686531 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.693913 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.693928 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.695116 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.695123 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695191 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695196 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695199 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:39:09.696606 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695202 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695205 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695208 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695211 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695214 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695217 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695220 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695222 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695225 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695234 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695237 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695240 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695242 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695245 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695249 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695253 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695256 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695258 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695272 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:39:09.696963 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695275 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695277 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695280 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695282 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695285 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695288 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695291 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695294 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695296 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695299 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695302 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695304 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695307 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695309 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695311 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695314 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695321 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695324 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695326 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695329 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:39:09.697441 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695331 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695334 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695336 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695344 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695347 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695349 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695352 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695354 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695357 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695359 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695363 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695367 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695370 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695372 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695375 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695378 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695380 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695383 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695386 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695389 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:39:09.697930 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695391 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695394 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695396 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695399 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695401 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695404 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695407 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695410 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695414 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695417 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695419 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695422 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695424 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695427 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695429 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695431 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695441 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695444 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695447 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:39:09.698440 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695449 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695452 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695454 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695457 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695460 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695462 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695465 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695467 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695470 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695472 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695474 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695477 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695479 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695482 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695484 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695487 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695489 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:39:09.698907 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695491 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.695496 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695658 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695663 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695666 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695670 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695673 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695675 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695678 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695681 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695684 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695686 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695689 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695700 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:39:09.699348 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695703 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695707 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695710 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695713 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695716 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695718 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695721 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695723 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695726 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695728 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695731 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695733 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695736 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695738 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695741 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695743 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695745 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695748 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695750 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:39:09.699708 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695753 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695755 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695757 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695760 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695762 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695765 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695767 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695770 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695772 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695774 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695777 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695779 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695782 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695790 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695793 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695796 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695798 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695801 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695803 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695806 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:39:09.700163 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695808 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695810 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695813 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695815 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695818 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695820 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695822 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695825 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695827 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695830 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695832 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695835 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695837 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695839 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695842 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695845 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695847 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695849 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695852 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695854 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:39:09.700658 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695857 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695859 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695862 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695864 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695867 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695869 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695877 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695880 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695882 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695885 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695887 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695890 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695892 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695895 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695897 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695900 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695902 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695905 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695907 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695909 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:39:09.701135 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695918 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:39:09.701654 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695920 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:39:09.701654 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695924 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:39:09.701654 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695927 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:39:09.701654 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695930 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:39:09.701654 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695933 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:39:09.701654 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695935 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:39:09.701654 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:09.695940 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:39:09.701654 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.695945 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:39:09.701654 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.696064 2556 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 17:39:09.701654 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.698404 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 17:39:09.701654 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.700068 2556 server.go:1019] "Starting client certificate rotation" Jul 10 17:39:09.701654 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.700174 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 17:39:09.701654 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.700242 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 17:39:09.704183 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.704127 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 17:39:09.708256 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.708235 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 17:39:09.716516 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.716499 2556 log.go:25] "Validated CRI v1 runtime API" Jul 10 17:39:09.719800 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.719786 2556 log.go:25] "Validated CRI v1 image API" Jul 10 17:39:09.720708 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.720695 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 17:39:09.723238 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.723218 2556 fs.go:135] Filesystem UUIDs: map[64dd3267-dedd-4cfa-8531-61e8da59685b:/dev/nvme0n1p3 6e848815-cc16-4dc4-a2de-c8043a15eb33:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 10 17:39:09.723301 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.723239 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 17:39:09.728968 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.728853 2556 manager.go:216] Machine: {Timestamp:2026-07-10 17:39:09.727470634 +0000 UTC m=+0.308501600 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103434 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21c3880d0d3699f3c916faad4c936e SystemUUID:ec21c388-0d0d-3699-f3c9-16faad4c936e BootID:c4aab96e-00b0-495d-81e8-2a98fd791656 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f1:d9:2b:9b:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f1:d9:2b:9b:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:38:05:4e:b6:b1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 17:39:09.728968 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.728961 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 17:39:09.729124 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.729045 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 17:39:09.730164 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.730146 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 17:39:09.732417 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.732391 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 17:39:09.732613 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.732423 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-103.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 17:39:09.733190 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.733178 2556 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 17:39:09.733242 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.733196 2556 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 17:39:09.733242 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.733207 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 17:39:09.733967 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.733954 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 17:39:09.735254 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.735242 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 17:39:09.738897 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.738884 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 10 17:39:09.739043 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.739032 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 17:39:09.739109 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.739081 2556 kubelet.go:486] "Attempting to sync node with API server" Jul 10 17:39:09.739109 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.739099 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 17:39:09.739200 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.739119 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 17:39:09.739200 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.739131 2556 kubelet.go:398] "Adding apiserver pod source" Jul 10 17:39:09.739200 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.739163 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 17:39:09.739611 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.739596 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 17:39:09.739670 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.739631 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 17:39:09.742177 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.742163 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 17:39:09.744207 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.744193 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 17:39:09.744249 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.744223 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 17:39:09.745901 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.745564 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 17:39:09.745986 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.745930 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 17:39:09.745986 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.745944 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 17:39:09.745986 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.745954 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 17:39:09.745986 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.745969 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 17:39:09.745986 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.745978 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 17:39:09.746166 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.745991 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 17:39:09.746166 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.746000 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 17:39:09.746166 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.746020 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 17:39:09.746166 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.746031 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 17:39:09.746166 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.746044 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 17:39:09.746166 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.746054 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 17:39:09.747048 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.747031 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 17:39:09.747118 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.747053 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 17:39:09.750751 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.750727 2556 server.go:1268] "Started kubelet" Jul 10 17:39:09.750847 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.750806 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 17:39:09.750930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.750877 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 17:39:09.750984 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.750954 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 17:39:09.751825 ip-10-0-137-103 systemd[1]: Started Kubernetes Kubelet. Jul 10 17:39:09.752832 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.752652 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 17:39:09.753028 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.753016 2556 server.go:310] "Adding debug handlers to kubelet server" Jul 10 17:39:09.754539 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.754509 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-103.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 17:39:09.754652 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.754627 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 17:39:09.754721 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.754707 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 17:39:09.757011 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.756958 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 17:39:09.757115 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.757042 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c29ng" Jul 10 17:39:09.758306 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.758254 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 17:39:09.758306 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.758282 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 17:39:09.758434 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.758368 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:09.759016 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.759003 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 17:39:09.759140 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.759116 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 17:39:09.759216 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.759027 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 17:39:09.759296 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.759227 2556 reconstruct.go:97] "Volume reconstruction finished" Jul 10 17:39:09.759296 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.759236 2556 reconciler.go:29] "Reconciler: start to sync state" Jul 10 17:39:09.759412 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.759397 2556 factory.go:55] Registering systemd factory Jul 10 17:39:09.759458 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.759419 2556 factory.go:223] Registration of the systemd container factory successfully Jul 10 17:39:09.759595 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.759582 2556 factory.go:153] Registering CRI-O factory Jul 10 17:39:09.759663 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.759596 2556 factory.go:223] Registration of the crio container factory successfully Jul 10 17:39:09.759663 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.759643 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 17:39:09.759663 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.759663 2556 factory.go:103] Registering Raw factory Jul 10 17:39:09.759772 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.759673 2556 manager.go:1196] Started watching for new ooms in manager Jul 10 17:39:09.759772 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.758825 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-103.ec2.internal.18c0fe621aecc19d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-103.ec2.internal,UID:ip-10-0-137-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-103.ec2.internal,},FirstTimestamp:2026-07-10 17:39:09.750690205 +0000 UTC m=+0.331721168,LastTimestamp:2026-07-10 17:39:09.750690205 +0000 UTC m=+0.331721168,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-103.ec2.internal,}" Jul 10 17:39:09.760147 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.760137 2556 manager.go:349] Starting recovery of all containers Jul 10 17:39:09.763859 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.763835 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 17:39:09.763944 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.763925 2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 17:39:09.764623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.764593 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c29ng" Jul 10 17:39:09.769232 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.769212 2556 manager.go:354] Recovery completed Jul 10 17:39:09.775354 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.775339 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:39:09.778491 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.778471 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:39:09.778589 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.778510 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:39:09.778589 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.778526 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:39:09.779093 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.779081 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:39:09.779154 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.779080 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 17:39:09.779154 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.779111 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 17:39:09.779154 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.779129 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 10 17:39:09.779845 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.779831 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:39:09.779910 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.779860 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:39:09.779910 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.779869 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:39:09.780368 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.780351 2556 policy_none.go:49] "None policy: Start" Jul 10 17:39:09.780413 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.780372 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 17:39:09.780413 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.780384 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 17:39:09.780870 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.780802 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-103.ec2.internal.18c0fe621c94fbc6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-103.ec2.internal,UID:ip-10-0-137-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-103.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-103.ec2.internal,},FirstTimestamp:2026-07-10 17:39:09.778492358 +0000 UTC m=+0.359523327,LastTimestamp:2026-07-10 17:39:09.778492358 +0000 UTC m=+0.359523327,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-103.ec2.internal,}" Jul 10 17:39:09.781398 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.781387 2556 policy_none.go:47] "Start" Jul 10 17:39:09.814931 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.814910 2556 manager.go:337] "Starting Device Plugin manager" Jul 10 17:39:09.815083 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.814945 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 17:39:09.815083 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.814955 2556 server.go:87] "Starting device plugin registration server" Jul 10 17:39:09.815247 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.815234 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 17:39:09.815302 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.815251 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 17:39:09.815410 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.815393 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 17:39:09.815506 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.815491 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 17:39:09.815588 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.815510 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 17:39:09.816005 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.815978 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 17:39:09.816089 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.816042 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:09.915717 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.915679 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:39:09.916920 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.916742 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:39:09.916920 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.916781 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:39:09.916920 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.916796 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:39:09.916920 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.916827 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-103.ec2.internal" Jul 10 17:39:09.922349 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.922332 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-103.ec2.internal" Jul 10 17:39:09.922401 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.922357 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-103.ec2.internal\": node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:09.938668 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.938600 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:09.940033 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.940013 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 17:39:09.941450 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.941422 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 17:39:09.941555 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.941479 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 17:39:09.941555 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.941515 2556 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 17:39:09.941658 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:09.941568 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 17:39:09.943584 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:09.943559 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:39:10.038748 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:10.038711 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:10.041871 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.041846 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-103.ec2.internal"] Jul 10 17:39:10.041938 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.041922 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:39:10.045173 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.045153 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:39:10.045286 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.045186 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:39:10.045286 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.045196 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:39:10.046565 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.046553 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:39:10.046673 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.046656 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.046718 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.046691 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:39:10.047462 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.047442 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:39:10.047563 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.047474 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:39:10.047563 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.047488 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:39:10.047563 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.047505 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:39:10.047563 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.047515 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:39:10.047563 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.047489 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:39:10.048709 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.048694 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.048775 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.048720 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:39:10.049637 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.049620 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:39:10.049727 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.049647 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:39:10.049727 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.049671 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:39:10.060212 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.060194 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5fecab31974d0fc256246ff6de1c24b9-config\") pod \"kube-apiserver-proxy-ip-10-0-137-103.ec2.internal\" (UID: \"5fecab31974d0fc256246ff6de1c24b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.060296 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.060220 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00178d55a78e9efa96676db1320af28c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal\" (UID: \"00178d55a78e9efa96676db1320af28c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.060296 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.060237 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00178d55a78e9efa96676db1320af28c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal\" (UID: \"00178d55a78e9efa96676db1320af28c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.071986 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:10.071965 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-103.ec2.internal\" not found" node="ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.075636 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:10.075621 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-103.ec2.internal\" not found" node="ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.139508 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:10.139472 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:10.160492 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.160456 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00178d55a78e9efa96676db1320af28c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal\" (UID: \"00178d55a78e9efa96676db1320af28c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.160492 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.160499 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5fecab31974d0fc256246ff6de1c24b9-config\") pod \"kube-apiserver-proxy-ip-10-0-137-103.ec2.internal\" (UID: \"5fecab31974d0fc256246ff6de1c24b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.160695 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.160531 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00178d55a78e9efa96676db1320af28c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal\" (UID: \"00178d55a78e9efa96676db1320af28c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.160695 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.160556 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00178d55a78e9efa96676db1320af28c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal\" (UID: \"00178d55a78e9efa96676db1320af28c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.160695 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.160566 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00178d55a78e9efa96676db1320af28c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal\" (UID: \"00178d55a78e9efa96676db1320af28c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.160695 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.160571 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5fecab31974d0fc256246ff6de1c24b9-config\") pod \"kube-apiserver-proxy-ip-10-0-137-103.ec2.internal\" (UID: \"5fecab31974d0fc256246ff6de1c24b9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.239802 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:10.239731 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:10.340378 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:10.340340 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:10.375878 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.375849 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.379010 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.378988 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-103.ec2.internal" Jul 10 17:39:10.441134 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:10.441096 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:10.541703 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:10.541626 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:10.595183 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.595148 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:39:10.642277 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:10.642238 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:10.699736 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.699702 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 17:39:10.700324 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.699866 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 17:39:10.700324 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:10.699894 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 17:39:10.743207 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:10.743173 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:12.079667 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:12.079637 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 17:34:11 +0000 UTC" deadline="2028-01-24 17:12:49.587457924 +0000 UTC" Jul 10 17:39:12.079730 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:12.079667 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13511h33m37.507794467s" Jul 10 17:39:12.150938 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:12.150862 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:12.225516 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:12.225492 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:39:12.251616 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:12.251594 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:12.273636 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:12.273602 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00178d55a78e9efa96676db1320af28c.slice/crio-9db135615ebde30fd4b44c4400cba1cfea565df4eb233fbb28410dde16863ade WatchSource:0}: Error finding container 9db135615ebde30fd4b44c4400cba1cfea565df4eb233fbb28410dde16863ade: Status 404 returned error can't find the container with id 9db135615ebde30fd4b44c4400cba1cfea565df4eb233fbb28410dde16863ade Jul 10 17:39:12.273858 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:12.273831 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fecab31974d0fc256246ff6de1c24b9.slice/crio-6ec33bec0b433ef0794cb905671031722784017a9f47fd71a0e0065aed8e7c74 WatchSource:0}: Error finding container 6ec33bec0b433ef0794cb905671031722784017a9f47fd71a0e0065aed8e7c74: Status 404 returned error can't find the container with id 6ec33bec0b433ef0794cb905671031722784017a9f47fd71a0e0065aed8e7c74 Jul 10 17:39:12.278673 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:12.278659 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:39:12.352233 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:12.352194 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:12.452815 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:12.452736 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:12.553577 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:12.553537 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-103.ec2.internal\" not found" Jul 10 17:39:12.586464 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:12.586437 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:39:12.665518 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:12.665490 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal" Jul 10 17:39:12.676010 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:12.675945 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 17:39:12.676912 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:12.676898 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-103.ec2.internal" Jul 10 17:39:12.685152 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:12.685131 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 17:39:13.046984 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.046935 2556 apiserver.go:52] "Watching apiserver" Jul 10 17:39:13.055129 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.054881 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 17:39:13.056171 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.056146 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-103.ec2.internal","openshift-image-registry/node-ca-4sdqt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-x9rhq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5","openshift-cluster-node-tuning-operator/tuned-7rszq","openshift-dns/node-resolver-flzpz","openshift-multus/multus-additional-cni-plugins-ntwdg","openshift-multus/multus-ln7km","openshift-multus/network-metrics-daemon-p58j7","openshift-network-diagnostics/network-check-target-t9t5m","openshift-network-operator/iptables-alerter-9zz5s","kube-system/global-pull-secret-syncer-wnspm","kube-system/konnectivity-agent-v7gd5"] Jul 10 17:39:13.058046 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.057614 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-flzpz" Jul 10 17:39:13.058705 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.058686 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:13.058824 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.058763 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:13.060298 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.060052 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 17:39:13.060298 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.060055 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-glwbs\"" Jul 10 17:39:13.060298 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.060224 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 17:39:13.061383 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.061366 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v7gd5" Jul 10 17:39:13.062493 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.062471 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.063807 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.063449 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.063807 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.063605 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 17:39:13.063996 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.063982 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 17:39:13.064068 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.064035 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6l57c\"" Jul 10 17:39:13.064726 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.064696 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.064895 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.064872 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 17:39:13.065084 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.065067 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 17:39:13.065280 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.065073 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 17:39:13.065351 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.065278 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 17:39:13.065351 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.065301 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 17:39:13.065576 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.065262 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nswpd\"" Jul 10 17:39:13.065576 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.065398 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 17:39:13.065690 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.065672 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 17:39:13.065794 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.065778 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 17:39:13.066155 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.066137 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9rnfm\"" Jul 10 17:39:13.066232 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.066188 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 17:39:13.066293 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.066250 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 17:39:13.066718 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.066382 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.067019 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.066995 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:39:13.067351 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.067333 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.067585 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.067563 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 17:39:13.067659 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.067601 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6rf7x\"" Jul 10 17:39:13.068529 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.068507 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 17:39:13.068762 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.068745 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 17:39:13.068878 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.068861 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 17:39:13.068944 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.068929 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nkqhq\"" Jul 10 17:39:13.069027 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.068982 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4sdqt" Jul 10 17:39:13.069171 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.068931 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 17:39:13.069402 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.069385 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 17:39:13.069876 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.069860 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:13.069942 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.069894 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dp9rw\"" Jul 10 17:39:13.069942 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.069923 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:13.070216 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.070198 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 17:39:13.071169 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.071149 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 17:39:13.071271 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.071213 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 17:39:13.071271 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.071216 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 17:39:13.071389 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.071373 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z8qz6\"" Jul 10 17:39:13.071577 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.071542 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:13.071647 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.071632 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:13.072872 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.072814 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9zz5s" Jul 10 17:39:13.075766 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.075204 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:39:13.076766 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.076649 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 17:39:13.076866 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.076839 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gdrv8\"" Jul 10 17:39:13.077167 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077148 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9nxm\" (UniqueName: \"kubernetes.io/projected/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-kube-api-access-r9nxm\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.077252 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077175 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-cni-bin\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.077307 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077257 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czlnp\" (UniqueName: \"kubernetes.io/projected/a57b3765-cc4f-407c-b815-f1c5958e66e8-kube-api-access-czlnp\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.077307 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077276 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-run-k8s-cni-cncf-io\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.077415 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077322 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-var-lib-kubelet\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.077415 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077359 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-tmp\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.077505 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077466 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-slash\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.077552 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077506 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-run-netns\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.077552 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077537 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-var-lib-cni-multus\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.077654 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077562 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-etc-kubernetes\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.077654 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077604 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-os-release\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.077739 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077653 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-log-socket\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.077739 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077676 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f9pb\" (UniqueName: \"kubernetes.io/projected/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-kube-api-access-6f9pb\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.077739 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077705 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-system-cni-dir\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.077739 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077733 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-tuned\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.077915 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077762 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-run-openvswitch\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.077915 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077789 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-env-overrides\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.077915 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077811 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-ovnkube-script-lib\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.077915 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077839 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-cni-binary-copy\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.077915 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077867 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spcpx\" (UniqueName: \"kubernetes.io/projected/3a04a38b-bb00-4ab0-b1d2-433b32e982e2-kube-api-access-spcpx\") pod \"node-resolver-flzpz\" (UID: \"3a04a38b-bb00-4ab0-b1d2-433b32e982e2\") " pod="openshift-dns/node-resolver-flzpz" Jul 10 17:39:13.077915 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077893 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-var-lib-openvswitch\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.078226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077918 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-node-log\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.078226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.077943 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.078226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078024 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-socket-dir\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.078226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078062 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-device-dir\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.078226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078094 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-kubernetes\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.078226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078141 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-cni-netd\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.078226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078165 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-multus-cni-dir\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.078226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078200 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-multus-conf-dir\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.078606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078254 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-multus-daemon-config\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.078606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078287 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a04a38b-bb00-4ab0-b1d2-433b32e982e2-hosts-file\") pod \"node-resolver-flzpz\" (UID: \"3a04a38b-bb00-4ab0-b1d2-433b32e982e2\") " pod="openshift-dns/node-resolver-flzpz" Jul 10 17:39:13.078606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078323 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a04a38b-bb00-4ab0-b1d2-433b32e982e2-tmp-dir\") pod \"node-resolver-flzpz\" (UID: \"3a04a38b-bb00-4ab0-b1d2-433b32e982e2\") " pod="openshift-dns/node-resolver-flzpz" Jul 10 17:39:13.078606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078364 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.078606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078408 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.078606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078439 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-ovn-node-metrics-cert\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.078606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078504 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-sysctl-conf\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.078606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078534 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-run-multus-certs\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.078606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078558 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-modprobe-d\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.078606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078586 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-sysconfig\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.079059 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078649 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-sysctl-d\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.079059 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078693 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv8gx\" (UniqueName: \"kubernetes.io/projected/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-kube-api-access-hv8gx\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.079059 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078724 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-run-systemd\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.079059 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078763 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-ovnkube-config\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.079059 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078795 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-registration-dir\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.079059 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078820 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-lib-modules\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.079059 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078849 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6828147f-7c39-49c2-b67b-284254fe0357-konnectivity-ca\") pod \"konnectivity-agent-v7gd5\" (UID: \"6828147f-7c39-49c2-b67b-284254fe0357\") " pod="kube-system/konnectivity-agent-v7gd5" Jul 10 17:39:13.079059 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078877 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-run-netns\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.079059 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078906 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-systemd\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.079059 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078933 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-sys\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.079059 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.078960 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.079059 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079032 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-systemd-units\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079067 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-etc-openvswitch\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079096 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/779679cf-3637-4afa-a580-767d44eb5e09-dbus\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079125 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-run\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079147 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6828147f-7c39-49c2-b67b-284254fe0357-agent-certs\") pod \"konnectivity-agent-v7gd5\" (UID: \"6828147f-7c39-49c2-b67b-284254fe0357\") " pod="kube-system/konnectivity-agent-v7gd5" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079175 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-os-release\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079202 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzf4r\" (UniqueName: \"kubernetes.io/projected/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-kube-api-access-xzf4r\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079229 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/779679cf-3637-4afa-a580-767d44eb5e09-kubelet-config\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079263 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-run-ovn\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079286 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-run-ovn-kubernetes\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079313 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-cnibin\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079388 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-var-lib-cni-bin\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079488 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-var-lib-kubelet\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079540 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-hostroot\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.079632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079583 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-host\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.080250 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079628 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-cni-binary-copy\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.080250 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079706 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.080250 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079780 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.080250 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079824 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-multus-socket-dir-parent\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.080250 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079881 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-system-cni-dir\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.080250 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079917 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-cnibin\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.080250 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.079946 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-kubelet\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.080250 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.080039 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-sys-fs\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.080250 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.080059 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 17:39:13.080250 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.080079 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:13.081478 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.081456 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 17:39:13.111397 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.111375 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rmmps" Jul 10 17:39:13.121895 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.121167 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rmmps" Jul 10 17:39:13.167783 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.167756 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 17:39:13.180607 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180577 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-socket-dir\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.180773 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180617 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-device-dir\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.180773 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180643 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-kubernetes\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.180773 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180668 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-cni-netd\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.180773 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180691 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-multus-cni-dir\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.180773 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180724 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-multus-conf-dir\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.180773 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180746 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-multus-daemon-config\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180779 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a04a38b-bb00-4ab0-b1d2-433b32e982e2-hosts-file\") pod \"node-resolver-flzpz\" (UID: \"3a04a38b-bb00-4ab0-b1d2-433b32e982e2\") " pod="openshift-dns/node-resolver-flzpz" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180806 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-socket-dir\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a04a38b-bb00-4ab0-b1d2-433b32e982e2-tmp-dir\") pod \"node-resolver-flzpz\" (UID: \"3a04a38b-bb00-4ab0-b1d2-433b32e982e2\") " pod="openshift-dns/node-resolver-flzpz" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180864 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180887 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180905 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-ovn-node-metrics-cert\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180921 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-sysctl-conf\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180935 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-run-multus-certs\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180954 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bntfx\" (UniqueName: \"kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx\") pod \"network-check-target-t9t5m\" (UID: \"cb66a47f-88eb-4ec2-a133-b8c5c85eed78\") " pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.180996 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-modprobe-d\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181014 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-sysconfig\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181039 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-sysctl-d\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv8gx\" (UniqueName: \"kubernetes.io/projected/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-kube-api-access-hv8gx\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181074 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-run-systemd\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181093 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-ovnkube-config\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181107 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-registration-dir\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.181137 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181123 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-lib-modules\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181138 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6828147f-7c39-49c2-b67b-284254fe0357-konnectivity-ca\") pod \"konnectivity-agent-v7gd5\" (UID: \"6828147f-7c39-49c2-b67b-284254fe0357\") " pod="kube-system/konnectivity-agent-v7gd5" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181152 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-run-netns\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181170 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93707f3d-99ad-44b3-9f3d-0d6ed8773dc4-iptables-alerter-script\") pod \"iptables-alerter-9zz5s\" (UID: \"93707f3d-99ad-44b3-9f3d-0d6ed8773dc4\") " pod="openshift-network-operator/iptables-alerter-9zz5s" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181185 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-systemd\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181199 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-sys\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181213 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181247 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-systemd-units\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181269 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-etc-openvswitch\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181288 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-device-dir\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181292 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/779679cf-3637-4afa-a580-767d44eb5e09-dbus\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181331 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r44w\" (UniqueName: \"kubernetes.io/projected/93707f3d-99ad-44b3-9f3d-0d6ed8773dc4-kube-api-access-2r44w\") pod \"iptables-alerter-9zz5s\" (UID: \"93707f3d-99ad-44b3-9f3d-0d6ed8773dc4\") " pod="openshift-network-operator/iptables-alerter-9zz5s" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181355 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181378 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-run\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181399 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6828147f-7c39-49c2-b67b-284254fe0357-agent-certs\") pod \"konnectivity-agent-v7gd5\" (UID: \"6828147f-7c39-49c2-b67b-284254fe0357\") " pod="kube-system/konnectivity-agent-v7gd5" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181427 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/779679cf-3637-4afa-a580-767d44eb5e09-dbus\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181433 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-os-release\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.181623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181458 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzf4r\" (UniqueName: \"kubernetes.io/projected/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-kube-api-access-xzf4r\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181484 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/779679cf-3637-4afa-a580-767d44eb5e09-kubelet-config\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181505 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67a35b3c-e9c2-45f7-9518-9754c6015eb0-host\") pod \"node-ca-4sdqt\" (UID: \"67a35b3c-e9c2-45f7-9518-9754c6015eb0\") " pod="openshift-image-registry/node-ca-4sdqt" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181547 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67a35b3c-e9c2-45f7-9518-9754c6015eb0-serviceca\") pod \"node-ca-4sdqt\" (UID: \"67a35b3c-e9c2-45f7-9518-9754c6015eb0\") " pod="openshift-image-registry/node-ca-4sdqt" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181574 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-run-ovn\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181600 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-run-ovn-kubernetes\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181627 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-cnibin\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181664 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-var-lib-cni-bin\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181695 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-var-lib-kubelet\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181721 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-hostroot\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181741 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-host\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181766 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-cni-binary-copy\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181792 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181846 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a04a38b-bb00-4ab0-b1d2-433b32e982e2-tmp-dir\") pod \"node-resolver-flzpz\" (UID: \"3a04a38b-bb00-4ab0-b1d2-433b32e982e2\") " pod="openshift-dns/node-resolver-flzpz" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181862 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-multus-socket-dir-parent\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181901 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-system-cni-dir\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.182159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181919 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-cnibin\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181925 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-multus-socket-dir-parent\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181937 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-kubelet\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181953 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-sys-fs\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.181987 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182004 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-kubernetes\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182014 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182049 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-kubelet\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182066 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-sys-fs\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182069 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-system-cni-dir\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182154 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-run-ovn\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182168 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-run-ovn-kubernetes\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182207 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-cnibin\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182004 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9nxm\" (UniqueName: \"kubernetes.io/projected/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-kube-api-access-r9nxm\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182230 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182243 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-cni-bin\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182271 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czlnp\" (UniqueName: \"kubernetes.io/projected/a57b3765-cc4f-407c-b815-f1c5958e66e8-kube-api-access-czlnp\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.182930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182299 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-run-k8s-cni-cncf-io\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182328 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f8xb\" (UniqueName: \"kubernetes.io/projected/17b57439-069e-42d9-b59f-976e14670388-kube-api-access-5f8xb\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182376 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-cnibin\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182384 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-run\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182419 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-var-lib-kubelet\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182452 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-tmp\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182482 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-slash\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182508 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-run-netns\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182535 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-var-lib-cni-multus\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182561 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-etc-kubernetes\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182587 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-os-release\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182614 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-log-socket\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182628 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-ovnkube-config\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182641 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f9pb\" (UniqueName: \"kubernetes.io/projected/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-kube-api-access-6f9pb\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182646 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6828147f-7c39-49c2-b67b-284254fe0357-konnectivity-ca\") pod \"konnectivity-agent-v7gd5\" (UID: \"6828147f-7c39-49c2-b67b-284254fe0357\") " pod="kube-system/konnectivity-agent-v7gd5" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182687 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-system-cni-dir\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182703 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-run-k8s-cni-cncf-io\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.182715 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:13.183728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182719 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93707f3d-99ad-44b3-9f3d-0d6ed8773dc4-host-slash\") pod \"iptables-alerter-9zz5s\" (UID: \"93707f3d-99ad-44b3-9f3d-0d6ed8773dc4\") " pod="openshift-network-operator/iptables-alerter-9zz5s" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182755 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182763 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-registration-dir\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182772 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-run-netns\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.182780 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret podName:779679cf-3637-4afa-a580-767d44eb5e09 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:13.682758435 +0000 UTC m=+2.957085128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret") pod "global-pull-secret-syncer-wnspm" (UID: "779679cf-3637-4afa-a580-767d44eb5e09") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182795 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-systemd\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182819 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-var-lib-kubelet\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182876 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-sysctl-d\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182440 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-cni-bin\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183060 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-slash\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183088 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-cni-binary-copy\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183100 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-run-netns\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.182720 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183119 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183163 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-tuned\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183178 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/779679cf-3637-4afa-a580-767d44eb5e09-kubelet-config\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183185 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-var-lib-kubelet\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.184405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183197 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-run-openvswitch\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183223 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-var-lib-cni-bin\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183223 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-system-cni-dir\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183240 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-env-overrides\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183243 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-lib-modules\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183105 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-os-release\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183270 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-run-systemd\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183148 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-hostroot\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183330 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-sys\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183334 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-multus-cni-dir\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183349 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-log-socket\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183360 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-systemd-units\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183380 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-sysctl-conf\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183408 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-ovnkube-script-lib\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183437 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-host-cni-netd\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183459 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-etc-openvswitch\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183443 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-cni-binary-copy\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183502 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spcpx\" (UniqueName: \"kubernetes.io/projected/3a04a38b-bb00-4ab0-b1d2-433b32e982e2-kube-api-access-spcpx\") pod \"node-resolver-flzpz\" (UID: \"3a04a38b-bb00-4ab0-b1d2-433b32e982e2\") " pod="openshift-dns/node-resolver-flzpz" Jul 10 17:39:13.184937 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183513 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-os-release\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183516 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-run-multus-certs\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183529 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bfc7\" (UniqueName: \"kubernetes.io/projected/67a35b3c-e9c2-45f7-9518-9754c6015eb0-kube-api-access-6bfc7\") pod \"node-ca-4sdqt\" (UID: \"67a35b3c-e9c2-45f7-9518-9754c6015eb0\") " pod="openshift-image-registry/node-ca-4sdqt" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183544 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-run-openvswitch\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183556 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-var-lib-openvswitch\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183559 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-host-var-lib-cni-multus\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183410 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-sysconfig\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183146 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-host\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183620 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-multus-conf-dir\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183623 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-etc-kubernetes\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183648 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-var-lib-openvswitch\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183703 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-modprobe-d\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183720 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-env-overrides\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.183752 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a04a38b-bb00-4ab0-b1d2-433b32e982e2-hosts-file\") pod \"node-resolver-flzpz\" (UID: \"3a04a38b-bb00-4ab0-b1d2-433b32e982e2\") " pod="openshift-dns/node-resolver-flzpz" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.184025 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-node-log\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.184087 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.184222 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-node-log\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.184579 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.186050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.184591 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a57b3765-cc4f-407c-b815-f1c5958e66e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.187007 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.184618 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-multus-daemon-config\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.187007 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.184935 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-cni-binary-copy\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.187007 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.185226 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-ovnkube-script-lib\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.187007 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.186829 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-tmp\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.188005 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.187751 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-ovn-node-metrics-cert\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.188005 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.187874 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6828147f-7c39-49c2-b67b-284254fe0357-agent-certs\") pod \"konnectivity-agent-v7gd5\" (UID: \"6828147f-7c39-49c2-b67b-284254fe0357\") " pod="kube-system/konnectivity-agent-v7gd5" Jul 10 17:39:13.189416 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.189350 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-etc-tuned\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.192782 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.192759 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czlnp\" (UniqueName: \"kubernetes.io/projected/a57b3765-cc4f-407c-b815-f1c5958e66e8-kube-api-access-czlnp\") pod \"aws-ebs-csi-driver-node-h4sr5\" (UID: \"a57b3765-cc4f-407c-b815-f1c5958e66e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.198881 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.197144 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f9pb\" (UniqueName: \"kubernetes.io/projected/ac5b01b9-68ea-422b-a5a4-48b6a59d48df-kube-api-access-6f9pb\") pod \"ovnkube-node-x9rhq\" (UID: \"ac5b01b9-68ea-422b-a5a4-48b6a59d48df\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.198881 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.197472 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spcpx\" (UniqueName: \"kubernetes.io/projected/3a04a38b-bb00-4ab0-b1d2-433b32e982e2-kube-api-access-spcpx\") pod \"node-resolver-flzpz\" (UID: \"3a04a38b-bb00-4ab0-b1d2-433b32e982e2\") " pod="openshift-dns/node-resolver-flzpz" Jul 10 17:39:13.199323 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.199301 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv8gx\" (UniqueName: \"kubernetes.io/projected/6c1e57d3-ce03-46d2-9e55-2768d908dbb0-kube-api-access-hv8gx\") pod \"tuned-7rszq\" (UID: \"6c1e57d3-ce03-46d2-9e55-2768d908dbb0\") " pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.199406 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.199365 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzf4r\" (UniqueName: \"kubernetes.io/projected/0ca99128-dbcc-4b37-98d3-4d9d70934cc8-kube-api-access-xzf4r\") pod \"multus-ln7km\" (UID: \"0ca99128-dbcc-4b37-98d3-4d9d70934cc8\") " pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.201896 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.201868 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9nxm\" (UniqueName: \"kubernetes.io/projected/63f4fcc3-2ddc-40fa-8a79-7f02580ce489-kube-api-access-r9nxm\") pod \"multus-additional-cni-plugins-ntwdg\" (UID: \"63f4fcc3-2ddc-40fa-8a79-7f02580ce489\") " pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.253871 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.253819 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-103.ec2.internal" event={"ID":"5fecab31974d0fc256246ff6de1c24b9","Type":"ContainerStarted","Data":"6ec33bec0b433ef0794cb905671031722784017a9f47fd71a0e0065aed8e7c74"} Jul 10 17:39:13.255276 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.255250 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal" event={"ID":"00178d55a78e9efa96676db1320af28c","Type":"ContainerStarted","Data":"9db135615ebde30fd4b44c4400cba1cfea565df4eb233fbb28410dde16863ade"} Jul 10 17:39:13.284627 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.284587 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfc7\" (UniqueName: \"kubernetes.io/projected/67a35b3c-e9c2-45f7-9518-9754c6015eb0-kube-api-access-6bfc7\") pod \"node-ca-4sdqt\" (UID: \"67a35b3c-e9c2-45f7-9518-9754c6015eb0\") " pod="openshift-image-registry/node-ca-4sdqt" Jul 10 17:39:13.284773 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.284650 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntfx\" (UniqueName: \"kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx\") pod \"network-check-target-t9t5m\" (UID: \"cb66a47f-88eb-4ec2-a133-b8c5c85eed78\") " pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:13.284773 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.284680 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93707f3d-99ad-44b3-9f3d-0d6ed8773dc4-iptables-alerter-script\") pod \"iptables-alerter-9zz5s\" (UID: \"93707f3d-99ad-44b3-9f3d-0d6ed8773dc4\") " pod="openshift-network-operator/iptables-alerter-9zz5s" Jul 10 17:39:13.284773 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.284707 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r44w\" (UniqueName: \"kubernetes.io/projected/93707f3d-99ad-44b3-9f3d-0d6ed8773dc4-kube-api-access-2r44w\") pod \"iptables-alerter-9zz5s\" (UID: \"93707f3d-99ad-44b3-9f3d-0d6ed8773dc4\") " pod="openshift-network-operator/iptables-alerter-9zz5s" Jul 10 17:39:13.284773 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.284731 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:13.284773 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.284761 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67a35b3c-e9c2-45f7-9518-9754c6015eb0-host\") pod \"node-ca-4sdqt\" (UID: \"67a35b3c-e9c2-45f7-9518-9754c6015eb0\") " pod="openshift-image-registry/node-ca-4sdqt" Jul 10 17:39:13.285054 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.284784 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67a35b3c-e9c2-45f7-9518-9754c6015eb0-serviceca\") pod \"node-ca-4sdqt\" (UID: \"67a35b3c-e9c2-45f7-9518-9754c6015eb0\") " pod="openshift-image-registry/node-ca-4sdqt" Jul 10 17:39:13.285054 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.284956 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:13.285054 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.284951 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f8xb\" (UniqueName: \"kubernetes.io/projected/17b57439-069e-42d9-b59f-976e14670388-kube-api-access-5f8xb\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:13.285054 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.285040 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs podName:17b57439-069e-42d9-b59f-976e14670388 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:13.785020164 +0000 UTC m=+3.059346854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs") pod "network-metrics-daemon-p58j7" (UID: "17b57439-069e-42d9-b59f-976e14670388") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:13.285261 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.285062 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67a35b3c-e9c2-45f7-9518-9754c6015eb0-host\") pod \"node-ca-4sdqt\" (UID: \"67a35b3c-e9c2-45f7-9518-9754c6015eb0\") " pod="openshift-image-registry/node-ca-4sdqt" Jul 10 17:39:13.285261 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.285078 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93707f3d-99ad-44b3-9f3d-0d6ed8773dc4-host-slash\") pod \"iptables-alerter-9zz5s\" (UID: \"93707f3d-99ad-44b3-9f3d-0d6ed8773dc4\") " pod="openshift-network-operator/iptables-alerter-9zz5s" Jul 10 17:39:13.285261 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.285147 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93707f3d-99ad-44b3-9f3d-0d6ed8773dc4-host-slash\") pod \"iptables-alerter-9zz5s\" (UID: \"93707f3d-99ad-44b3-9f3d-0d6ed8773dc4\") " pod="openshift-network-operator/iptables-alerter-9zz5s" Jul 10 17:39:13.285409 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.285283 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67a35b3c-e9c2-45f7-9518-9754c6015eb0-serviceca\") pod \"node-ca-4sdqt\" (UID: \"67a35b3c-e9c2-45f7-9518-9754c6015eb0\") " pod="openshift-image-registry/node-ca-4sdqt" Jul 10 17:39:13.285409 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.285364 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93707f3d-99ad-44b3-9f3d-0d6ed8773dc4-iptables-alerter-script\") pod \"iptables-alerter-9zz5s\" (UID: \"93707f3d-99ad-44b3-9f3d-0d6ed8773dc4\") " pod="openshift-network-operator/iptables-alerter-9zz5s" Jul 10 17:39:13.293566 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.293540 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:39:13.293566 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.293571 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:39:13.293778 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.293587 2556 projected.go:196] Error preparing data for projected volume kube-api-access-bntfx for pod openshift-network-diagnostics/network-check-target-t9t5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:13.293778 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.293663 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx podName:cb66a47f-88eb-4ec2-a133-b8c5c85eed78 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:13.793644446 +0000 UTC m=+3.067971122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bntfx" (UniqueName: "kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx") pod "network-check-target-t9t5m" (UID: "cb66a47f-88eb-4ec2-a133-b8c5c85eed78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:13.295507 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.295485 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f8xb\" (UniqueName: \"kubernetes.io/projected/17b57439-069e-42d9-b59f-976e14670388-kube-api-access-5f8xb\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:13.296385 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.296363 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r44w\" (UniqueName: \"kubernetes.io/projected/93707f3d-99ad-44b3-9f3d-0d6ed8773dc4-kube-api-access-2r44w\") pod \"iptables-alerter-9zz5s\" (UID: \"93707f3d-99ad-44b3-9f3d-0d6ed8773dc4\") " pod="openshift-network-operator/iptables-alerter-9zz5s" Jul 10 17:39:13.299060 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.299010 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bfc7\" (UniqueName: \"kubernetes.io/projected/67a35b3c-e9c2-45f7-9518-9754c6015eb0-kube-api-access-6bfc7\") pod \"node-ca-4sdqt\" (UID: \"67a35b3c-e9c2-45f7-9518-9754c6015eb0\") " pod="openshift-image-registry/node-ca-4sdqt" Jul 10 17:39:13.371146 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.371116 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-flzpz" Jul 10 17:39:13.381517 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.381491 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v7gd5" Jul 10 17:39:13.391296 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.391272 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:13.397020 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.396999 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" Jul 10 17:39:13.405590 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.405569 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7rszq" Jul 10 17:39:13.414245 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.414225 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ln7km" Jul 10 17:39:13.422947 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.422929 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ntwdg" Jul 10 17:39:13.433563 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.433539 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4sdqt" Jul 10 17:39:13.441294 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.441276 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9zz5s" Jul 10 17:39:13.448435 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.448412 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:39:13.687852 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.687816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:13.688027 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.687954 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:13.688102 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.688036 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret podName:779679cf-3637-4afa-a580-767d44eb5e09 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:14.688015199 +0000 UTC m=+3.962341886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret") pod "global-pull-secret-syncer-wnspm" (UID: "779679cf-3637-4afa-a580-767d44eb5e09") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:13.788664 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.788631 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:13.788848 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.788817 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:13.788907 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.788896 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs podName:17b57439-069e-42d9-b59f-976e14670388 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:14.788876515 +0000 UTC m=+4.063203193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs") pod "network-metrics-daemon-p58j7" (UID: "17b57439-069e-42d9-b59f-976e14670388") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:13.795726 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.795697 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:39:13.889179 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:13.889097 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntfx\" (UniqueName: \"kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx\") pod \"network-check-target-t9t5m\" (UID: \"cb66a47f-88eb-4ec2-a133-b8c5c85eed78\") " pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:13.889321 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.889266 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:39:13.889321 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.889288 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:39:13.889321 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.889297 2556 projected.go:196] Error preparing data for projected volume kube-api-access-bntfx for pod openshift-network-diagnostics/network-check-target-t9t5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:13.889443 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:13.889350 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx podName:cb66a47f-88eb-4ec2-a133-b8c5c85eed78 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:14.889333632 +0000 UTC m=+4.163660306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bntfx" (UniqueName: "kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx") pod "network-check-target-t9t5m" (UID: "cb66a47f-88eb-4ec2-a133-b8c5c85eed78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:14.123260 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:14.123207 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 17:34:13 +0000 UTC" deadline="2028-03-01 15:16:33.719293174 +0000 UTC" Jul 10 17:39:14.123260 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:14.123256 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14397h37m19.59604307s" Jul 10 17:39:14.249407 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:14.249380 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:14.249502 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:14.249381 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:14.249545 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:14.249524 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:14.249634 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:14.249612 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:14.263527 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:14.263497 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1e57d3_ce03_46d2_9e55_2768d908dbb0.slice/crio-a4f1ab1bdbf026e2af609e36ef4df22cad6be246857c579dfd5d5414193468fa WatchSource:0}: Error finding container a4f1ab1bdbf026e2af609e36ef4df22cad6be246857c579dfd5d5414193468fa: Status 404 returned error can't find the container with id a4f1ab1bdbf026e2af609e36ef4df22cad6be246857c579dfd5d5414193468fa Jul 10 17:39:14.265456 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:14.265394 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57b3765_cc4f_407c_b815_f1c5958e66e8.slice/crio-a6245847fd7819ce1ca5d61354470390a168d4eca91b420b7a8308cc7e62a9c8 WatchSource:0}: Error finding container a6245847fd7819ce1ca5d61354470390a168d4eca91b420b7a8308cc7e62a9c8: Status 404 returned error can't find the container with id a6245847fd7819ce1ca5d61354470390a168d4eca91b420b7a8308cc7e62a9c8 Jul 10 17:39:14.267616 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:14.267559 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca99128_dbcc_4b37_98d3_4d9d70934cc8.slice/crio-426da30fb0717f08c82c027c43f2df2ae95374a319ae5bd72496dfb9d359aea3 WatchSource:0}: Error finding container 426da30fb0717f08c82c027c43f2df2ae95374a319ae5bd72496dfb9d359aea3: Status 404 returned error can't find the container with id 426da30fb0717f08c82c027c43f2df2ae95374a319ae5bd72496dfb9d359aea3 Jul 10 17:39:14.270350 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:14.270201 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a04a38b_bb00_4ab0_b1d2_433b32e982e2.slice/crio-c1fb23ce2a4fe8e1bdb69ed45e00bd52ff5a860a450aa5684ce817ecc8da4a8f WatchSource:0}: Error finding container c1fb23ce2a4fe8e1bdb69ed45e00bd52ff5a860a450aa5684ce817ecc8da4a8f: Status 404 returned error can't find the container with id c1fb23ce2a4fe8e1bdb69ed45e00bd52ff5a860a450aa5684ce817ecc8da4a8f Jul 10 17:39:14.271398 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:14.271375 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a35b3c_e9c2_45f7_9518_9754c6015eb0.slice/crio-b82804faa216d7b2f80a3150a92d1a112766e7a8b4113e4e408d55aa640b4ed4 WatchSource:0}: Error finding container b82804faa216d7b2f80a3150a92d1a112766e7a8b4113e4e408d55aa640b4ed4: Status 404 returned error can't find the container with id b82804faa216d7b2f80a3150a92d1a112766e7a8b4113e4e408d55aa640b4ed4 Jul 10 17:39:14.272484 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:14.272444 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5b01b9_68ea_422b_a5a4_48b6a59d48df.slice/crio-3c4da8cd5c32c2ea9d52ce1efd67ef983772531c9d322b4a8974bfaf60e88959 WatchSource:0}: Error finding container 3c4da8cd5c32c2ea9d52ce1efd67ef983772531c9d322b4a8974bfaf60e88959: Status 404 returned error can't find the container with id 3c4da8cd5c32c2ea9d52ce1efd67ef983772531c9d322b4a8974bfaf60e88959 Jul 10 17:39:14.273415 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:14.273323 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f4fcc3_2ddc_40fa_8a79_7f02580ce489.slice/crio-b0a34d55039ad64ed0da6a125d9ec945d70f28fecc60a5905d82a633a9226358 WatchSource:0}: Error finding container b0a34d55039ad64ed0da6a125d9ec945d70f28fecc60a5905d82a633a9226358: Status 404 returned error can't find the container with id b0a34d55039ad64ed0da6a125d9ec945d70f28fecc60a5905d82a633a9226358 Jul 10 17:39:14.274343 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:14.274322 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6828147f_7c39_49c2_b67b_284254fe0357.slice/crio-0d344097b52030d547a1ade3e4ef7ebc0a7adc8341807d528dd2fbdca3f47e41 WatchSource:0}: Error finding container 0d344097b52030d547a1ade3e4ef7ebc0a7adc8341807d528dd2fbdca3f47e41: Status 404 returned error can't find the container with id 0d344097b52030d547a1ade3e4ef7ebc0a7adc8341807d528dd2fbdca3f47e41 Jul 10 17:39:14.275675 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:14.275656 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93707f3d_99ad_44b3_9f3d_0d6ed8773dc4.slice/crio-25da368120fef96c45632fd5f9481cf90fe568d923a622bd22fa57709bb68afa WatchSource:0}: Error finding container 25da368120fef96c45632fd5f9481cf90fe568d923a622bd22fa57709bb68afa: Status 404 returned error can't find the container with id 25da368120fef96c45632fd5f9481cf90fe568d923a622bd22fa57709bb68afa Jul 10 17:39:14.695677 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:14.695470 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:14.695846 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:14.695625 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:14.695846 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:14.695782 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret podName:779679cf-3637-4afa-a580-767d44eb5e09 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:16.695766805 +0000 UTC m=+5.970093483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret") pod "global-pull-secret-syncer-wnspm" (UID: "779679cf-3637-4afa-a580-767d44eb5e09") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:14.796593 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:14.796497 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:14.796762 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:14.796678 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:14.796762 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:14.796743 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs podName:17b57439-069e-42d9-b59f-976e14670388 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:16.796723897 +0000 UTC m=+6.071050583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs") pod "network-metrics-daemon-p58j7" (UID: "17b57439-069e-42d9-b59f-976e14670388") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:14.897757 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:14.897127 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntfx\" (UniqueName: \"kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx\") pod \"network-check-target-t9t5m\" (UID: \"cb66a47f-88eb-4ec2-a133-b8c5c85eed78\") " pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:14.897757 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:14.897337 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:39:14.897757 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:14.897359 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:39:14.897757 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:14.897371 2556 projected.go:196] Error preparing data for projected volume kube-api-access-bntfx for pod openshift-network-diagnostics/network-check-target-t9t5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:14.897757 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:14.897427 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx podName:cb66a47f-88eb-4ec2-a133-b8c5c85eed78 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:16.897408634 +0000 UTC m=+6.171735311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bntfx" (UniqueName: "kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx") pod "network-check-target-t9t5m" (UID: "cb66a47f-88eb-4ec2-a133-b8c5c85eed78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:15.124307 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.124147 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 17:34:13 +0000 UTC" deadline="2028-02-28 22:31:54.188191945 +0000 UTC" Jul 10 17:39:15.124307 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.124214 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14356h52m39.06398375s" Jul 10 17:39:15.251923 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.251400 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:15.251923 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:15.251537 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:15.271774 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.271735 2556 generic.go:356] "Generic (PLEG): container finished" podID="00178d55a78e9efa96676db1320af28c" containerID="b7522e339cb9ebadfcb6b950fe82f014cf636f515544fe5e0836737aec4e37c7" exitCode=0 Jul 10 17:39:15.271951 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.271834 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal" event={"ID":"00178d55a78e9efa96676db1320af28c","Type":"ContainerDied","Data":"b7522e339cb9ebadfcb6b950fe82f014cf636f515544fe5e0836737aec4e37c7"} Jul 10 17:39:15.289720 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.289510 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntwdg" event={"ID":"63f4fcc3-2ddc-40fa-8a79-7f02580ce489","Type":"ContainerStarted","Data":"b0a34d55039ad64ed0da6a125d9ec945d70f28fecc60a5905d82a633a9226358"} Jul 10 17:39:15.293978 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.293930 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" event={"ID":"ac5b01b9-68ea-422b-a5a4-48b6a59d48df","Type":"ContainerStarted","Data":"3c4da8cd5c32c2ea9d52ce1efd67ef983772531c9d322b4a8974bfaf60e88959"} Jul 10 17:39:15.304470 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.304136 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-flzpz" event={"ID":"3a04a38b-bb00-4ab0-b1d2-433b32e982e2","Type":"ContainerStarted","Data":"c1fb23ce2a4fe8e1bdb69ed45e00bd52ff5a860a450aa5684ce817ecc8da4a8f"} Jul 10 17:39:15.322954 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.322856 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ln7km" event={"ID":"0ca99128-dbcc-4b37-98d3-4d9d70934cc8","Type":"ContainerStarted","Data":"426da30fb0717f08c82c027c43f2df2ae95374a319ae5bd72496dfb9d359aea3"} Jul 10 17:39:15.329915 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.329812 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" event={"ID":"a57b3765-cc4f-407c-b815-f1c5958e66e8","Type":"ContainerStarted","Data":"a6245847fd7819ce1ca5d61354470390a168d4eca91b420b7a8308cc7e62a9c8"} Jul 10 17:39:15.337279 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.337196 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7rszq" event={"ID":"6c1e57d3-ce03-46d2-9e55-2768d908dbb0","Type":"ContainerStarted","Data":"a4f1ab1bdbf026e2af609e36ef4df22cad6be246857c579dfd5d5414193468fa"} Jul 10 17:39:15.362614 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.361893 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-103.ec2.internal" event={"ID":"5fecab31974d0fc256246ff6de1c24b9","Type":"ContainerStarted","Data":"bd8726ec461995b331e6bc502c587c70e20c1d31573d0e1c4f0dea346f2f7280"} Jul 10 17:39:15.376417 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.376306 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-103.ec2.internal" podStartSLOduration=3.376285371 podStartE2EDuration="3.376285371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:15.375178589 +0000 UTC m=+4.649505277" watchObservedRunningTime="2026-07-10 17:39:15.376285371 +0000 UTC m=+4.650612068" Jul 10 17:39:15.376651 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.376621 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9zz5s" event={"ID":"93707f3d-99ad-44b3-9f3d-0d6ed8773dc4","Type":"ContainerStarted","Data":"25da368120fef96c45632fd5f9481cf90fe568d923a622bd22fa57709bb68afa"} Jul 10 17:39:15.389341 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.389302 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v7gd5" event={"ID":"6828147f-7c39-49c2-b67b-284254fe0357","Type":"ContainerStarted","Data":"0d344097b52030d547a1ade3e4ef7ebc0a7adc8341807d528dd2fbdca3f47e41"} Jul 10 17:39:15.412928 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:15.412885 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4sdqt" event={"ID":"67a35b3c-e9c2-45f7-9518-9754c6015eb0","Type":"ContainerStarted","Data":"b82804faa216d7b2f80a3150a92d1a112766e7a8b4113e4e408d55aa640b4ed4"} Jul 10 17:39:16.248933 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:16.248901 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:16.249402 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:16.249052 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:16.249402 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:16.249150 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:16.249402 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:16.249257 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:16.435175 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:16.435137 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal" event={"ID":"00178d55a78e9efa96676db1320af28c","Type":"ContainerStarted","Data":"f90e6a5861992436781ef3bb0dc90dbef35b8fb593ba997ff58f71193b6df64c"} Jul 10 17:39:16.713800 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:16.713720 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:16.713989 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:16.713891 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:16.713989 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:16.713957 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret podName:779679cf-3637-4afa-a580-767d44eb5e09 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:20.713939332 +0000 UTC m=+9.988266008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret") pod "global-pull-secret-syncer-wnspm" (UID: "779679cf-3637-4afa-a580-767d44eb5e09") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:16.815051 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:16.815016 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:16.815649 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:16.815624 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:16.815750 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:16.815730 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs podName:17b57439-069e-42d9-b59f-976e14670388 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:20.815709351 +0000 UTC m=+10.090036027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs") pod "network-metrics-daemon-p58j7" (UID: "17b57439-069e-42d9-b59f-976e14670388") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:16.915986 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:16.915933 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntfx\" (UniqueName: \"kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx\") pod \"network-check-target-t9t5m\" (UID: \"cb66a47f-88eb-4ec2-a133-b8c5c85eed78\") " pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:16.916169 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:16.916114 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:39:16.916169 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:16.916136 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:39:16.916169 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:16.916149 2556 projected.go:196] Error preparing data for projected volume kube-api-access-bntfx for pod openshift-network-diagnostics/network-check-target-t9t5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:16.916322 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:16.916204 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx podName:cb66a47f-88eb-4ec2-a133-b8c5c85eed78 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:20.916186402 +0000 UTC m=+10.190513077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bntfx" (UniqueName: "kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx") pod "network-check-target-t9t5m" (UID: "cb66a47f-88eb-4ec2-a133-b8c5c85eed78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:17.252245 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:17.251103 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:17.252245 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:17.251256 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:18.248962 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:18.248922 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:18.249198 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:18.249080 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:18.249463 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:18.249445 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:18.249556 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:18.249519 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:19.251706 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:19.251060 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:19.251706 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:19.251201 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:20.248929 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:20.248893 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:20.249144 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:20.249049 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:20.249364 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:20.249347 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:20.249429 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:20.249415 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:20.751004 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:20.750943 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:20.751419 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:20.751093 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:20.751419 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:20.751153 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret podName:779679cf-3637-4afa-a580-767d44eb5e09 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:28.751133462 +0000 UTC m=+18.025460143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret") pod "global-pull-secret-syncer-wnspm" (UID: "779679cf-3637-4afa-a580-767d44eb5e09") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:20.851478 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:20.851436 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:20.851658 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:20.851610 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:20.851723 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:20.851690 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs podName:17b57439-069e-42d9-b59f-976e14670388 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:28.851670264 +0000 UTC m=+18.125996945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs") pod "network-metrics-daemon-p58j7" (UID: "17b57439-069e-42d9-b59f-976e14670388") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:20.952429 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:20.952315 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntfx\" (UniqueName: \"kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx\") pod \"network-check-target-t9t5m\" (UID: \"cb66a47f-88eb-4ec2-a133-b8c5c85eed78\") " pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:20.952609 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:20.952509 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:39:20.952609 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:20.952533 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:39:20.952609 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:20.952543 2556 projected.go:196] Error preparing data for projected volume kube-api-access-bntfx for pod openshift-network-diagnostics/network-check-target-t9t5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:20.952609 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:20.952603 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx podName:cb66a47f-88eb-4ec2-a133-b8c5c85eed78 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:28.952589399 +0000 UTC m=+18.226916072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bntfx" (UniqueName: "kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx") pod "network-check-target-t9t5m" (UID: "cb66a47f-88eb-4ec2-a133-b8c5c85eed78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:21.252803 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:21.252315 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:21.252803 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:21.252440 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:22.249022 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:22.248991 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:22.249022 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:22.249012 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:22.249553 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:22.249120 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:22.249553 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:22.249449 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:23.251445 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:23.251413 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:23.251885 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:23.251553 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:24.248833 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:24.248796 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:24.249050 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:24.248919 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:24.249257 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:24.249242 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:24.249360 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:24.249338 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:25.248878 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:25.248842 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:25.249348 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:25.249003 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:26.249102 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:26.249064 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:26.249571 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:26.249114 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:26.249571 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:26.249199 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:26.249571 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:26.249343 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:27.249292 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:27.249259 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:27.249807 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:27.249384 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:28.249217 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:28.249178 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:28.249507 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:28.249178 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:28.249507 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:28.249305 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:28.249507 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:28.249392 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:28.809554 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:28.809522 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:28.809748 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:28.809650 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:28.809748 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:28.809706 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret podName:779679cf-3637-4afa-a580-767d44eb5e09 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:44.809692399 +0000 UTC m=+34.084019073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret") pod "global-pull-secret-syncer-wnspm" (UID: "779679cf-3637-4afa-a580-767d44eb5e09") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:28.910049 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:28.910011 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:28.910208 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:28.910188 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:28.910272 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:28.910259 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs podName:17b57439-069e-42d9-b59f-976e14670388 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:44.910240477 +0000 UTC m=+34.184567153 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs") pod "network-metrics-daemon-p58j7" (UID: "17b57439-069e-42d9-b59f-976e14670388") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:29.011207 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:29.011169 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntfx\" (UniqueName: \"kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx\") pod \"network-check-target-t9t5m\" (UID: \"cb66a47f-88eb-4ec2-a133-b8c5c85eed78\") " pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:29.011365 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:29.011347 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:39:29.011428 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:29.011371 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:39:29.011428 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:29.011382 2556 projected.go:196] Error preparing data for projected volume kube-api-access-bntfx for pod openshift-network-diagnostics/network-check-target-t9t5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:29.011494 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:29.011447 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx podName:cb66a47f-88eb-4ec2-a133-b8c5c85eed78 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:45.011433078 +0000 UTC m=+34.285759751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bntfx" (UniqueName: "kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx") pod "network-check-target-t9t5m" (UID: "cb66a47f-88eb-4ec2-a133-b8c5c85eed78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:29.248892 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:29.248848 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:29.249086 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:29.249026 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:30.249080 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:30.249042 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:30.249559 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:30.249043 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:30.249559 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:30.249169 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:30.249559 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:30.249285 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:31.250384 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:31.250343 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:31.250916 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:31.250479 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:32.248620 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:32.248584 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:32.248798 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:32.248583 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:32.248798 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:32.248738 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:32.248798 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:32.248786 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:33.249269 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:33.249226 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:33.249744 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:33.249370 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:33.480630 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:33.480313 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ln7km" event={"ID":"0ca99128-dbcc-4b37-98d3-4d9d70934cc8","Type":"ContainerStarted","Data":"1823238c18ad3c5912c0d1cbd695f103d9d2b13e22e07da600296d381e5236bd"} Jul 10 17:39:33.483378 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:33.482826 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7rszq" event={"ID":"6c1e57d3-ce03-46d2-9e55-2768d908dbb0","Type":"ContainerStarted","Data":"d6811db5c0ebf4b5fb90c9c317416d84212ee4017f5c0952dc40b7639ecc35f4"} Jul 10 17:39:33.499925 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:33.499355 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-103.ec2.internal" podStartSLOduration=21.499340195 podStartE2EDuration="21.499340195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:16.449377135 +0000 UTC m=+5.723703830" watchObservedRunningTime="2026-07-10 17:39:33.499340195 +0000 UTC m=+22.773666909" Jul 10 17:39:33.499925 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:33.499720 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ln7km" podStartSLOduration=3.421653995 podStartE2EDuration="22.499712515s" totalImagesPullingTime="19.07805852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:33.498765135 +0000 UTC m=+22.773091869" watchObservedRunningTime="2026-07-10 17:39:33.499712515 +0000 UTC m=+22.774039210" Jul 10 17:39:33.523357 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:33.523312 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7rszq" podStartSLOduration=3.461482861 podStartE2EDuration="22.52329353s" totalImagesPullingTime="19.061810669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:33.5226487 +0000 UTC m=+22.796975420" watchObservedRunningTime="2026-07-10 17:39:33.52329353 +0000 UTC m=+22.797620225" Jul 10 17:39:34.248659 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.248460 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:34.248917 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.248512 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:34.248917 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:34.248753 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:34.248917 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:34.248800 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:34.489394 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.489356 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v7gd5" event={"ID":"6828147f-7c39-49c2-b67b-284254fe0357","Type":"ContainerStarted","Data":"d48b2b7072fd3af9eb91f93ffffaf36f47cef7ddea9df679a4fa96590dc00768"} Jul 10 17:39:34.490801 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.490776 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4sdqt" event={"ID":"67a35b3c-e9c2-45f7-9518-9754c6015eb0","Type":"ContainerStarted","Data":"30fd62c704266d2e367f39370f8aa88d9a0b26fc274b04a105ff5abb8c9ebe06"} Jul 10 17:39:34.492112 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.492087 2556 generic.go:356] "Generic (PLEG): container finished" podID="63f4fcc3-2ddc-40fa-8a79-7f02580ce489" containerID="a6c346afab9769e8c6cc38c1e032e53b52ee9830fe53bafd29f7d4904611df20" exitCode=0 Jul 10 17:39:34.492216 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.492155 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntwdg" event={"ID":"63f4fcc3-2ddc-40fa-8a79-7f02580ce489","Type":"ContainerDied","Data":"a6c346afab9769e8c6cc38c1e032e53b52ee9830fe53bafd29f7d4904611df20"} Jul 10 17:39:34.495136 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.495113 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" event={"ID":"ac5b01b9-68ea-422b-a5a4-48b6a59d48df","Type":"ContainerStarted","Data":"40783670541a818dc44f1963c85ff98076709c3e62040f9f1efced6142866c6a"} Jul 10 17:39:34.495243 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.495195 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" event={"ID":"ac5b01b9-68ea-422b-a5a4-48b6a59d48df","Type":"ContainerStarted","Data":"598b3369efd748ce9884c1e15791b829e279a59e89e448804ed303a8acad560c"} Jul 10 17:39:34.495299 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.495250 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" event={"ID":"ac5b01b9-68ea-422b-a5a4-48b6a59d48df","Type":"ContainerStarted","Data":"5df3f6ced85cd3b04f6dbc0fa1f6bfd2e7ae6d137e22c45a0cb41f5c34bfccb3"} Jul 10 17:39:34.495299 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.495285 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" event={"ID":"ac5b01b9-68ea-422b-a5a4-48b6a59d48df","Type":"ContainerStarted","Data":"379441b007d0575ea17162cef9a3a04089ffb0b003cf62da8ce6f978fa33b3be"} Jul 10 17:39:34.495385 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.495315 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" event={"ID":"ac5b01b9-68ea-422b-a5a4-48b6a59d48df","Type":"ContainerStarted","Data":"ac4833e0f818861967a5970f8e13fc22925cbd2c1a45519cd779c4f25505e050"} Jul 10 17:39:34.495385 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.495361 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" event={"ID":"ac5b01b9-68ea-422b-a5a4-48b6a59d48df","Type":"ContainerStarted","Data":"851599e2a1d7f4a8c8488ffe867ffb05cca8d75d151969feaf8f10a9da4a139c"} Jul 10 17:39:34.496400 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.496381 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-flzpz" event={"ID":"3a04a38b-bb00-4ab0-b1d2-433b32e982e2","Type":"ContainerStarted","Data":"79975b1e58a011d39e1ed82245be5c673b100b974e3064d041ae1d8deeb41afb"} Jul 10 17:39:34.497688 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.497668 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" event={"ID":"a57b3765-cc4f-407c-b815-f1c5958e66e8","Type":"ContainerStarted","Data":"9dbd18356a5580ae8a44c91e8f0ca52d3089a2cc6381d6e2880e47ec74857292"} Jul 10 17:39:34.505114 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.505074 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v7gd5" podStartSLOduration=4.482445586 podStartE2EDuration="23.505060591s" totalImagesPullingTime="19.022615005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:34.503594679 +0000 UTC m=+23.777921386" watchObservedRunningTime="2026-07-10 17:39:34.505060591 +0000 UTC m=+23.779387283" Jul 10 17:39:34.543552 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.543506 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4sdqt" podStartSLOduration=8.709451454 podStartE2EDuration="23.543488669s" totalImagesPullingTime="14.834037215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:34.543428082 +0000 UTC m=+23.817754777" watchObservedRunningTime="2026-07-10 17:39:34.543488669 +0000 UTC m=+23.817815364" Jul 10 17:39:34.562279 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:34.562217 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-flzpz" podStartSLOduration=4.512642811 podStartE2EDuration="23.562198079s" totalImagesPullingTime="19.049555268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:34.561821136 +0000 UTC m=+23.836147831" watchObservedRunningTime="2026-07-10 17:39:34.562198079 +0000 UTC m=+23.836524776" Jul 10 17:39:35.172072 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:35.171962 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 17:39:35.248839 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:35.248810 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:35.249053 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:35.248946 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:35.353065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:35.353031 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v7gd5" Jul 10 17:39:35.353801 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:35.353779 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v7gd5" Jul 10 17:39:35.502197 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:35.502163 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" event={"ID":"a57b3765-cc4f-407c-b815-f1c5958e66e8","Type":"ContainerStarted","Data":"553786c32bc167808870aef6ff7bfeee5327404c130142ab7f65614200f961a0"} Jul 10 17:39:35.503860 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:35.503825 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9zz5s" event={"ID":"93707f3d-99ad-44b3-9f3d-0d6ed8773dc4","Type":"ContainerStarted","Data":"a8639b9e0a66b15768ed1bd7ef34fab6448c1dc09add7a162311315ef28fad2b"} Jul 10 17:39:35.504474 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:35.504446 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v7gd5" Jul 10 17:39:35.505010 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:35.504992 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v7gd5" Jul 10 17:39:35.517701 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:35.517648 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9zz5s" podStartSLOduration=5.49820439 podStartE2EDuration="24.517628692s" totalImagesPullingTime="19.019424302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:35.517342776 +0000 UTC m=+24.791669471" watchObservedRunningTime="2026-07-10 17:39:35.517628692 +0000 UTC m=+24.791955388" Jul 10 17:39:36.136757 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:36.136659 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T17:39:35.172065523Z","UUID":"3cd7eec1-cc4e-4ed7-bb9f-73c9fd76eb42","Handler":null,"Name":"","Endpoint":""} Jul 10 17:39:36.138233 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:36.138213 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 17:39:36.138354 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:36.138267 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 17:39:36.248809 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:36.248780 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:36.249033 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:36.248904 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:36.249033 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:36.248949 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:36.249147 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:36.249058 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:36.511932 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:36.511896 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" event={"ID":"ac5b01b9-68ea-422b-a5a4-48b6a59d48df","Type":"ContainerStarted","Data":"8c14efe5154a3e09d9a3eee62087c7f96c398615e3b8daaf909e1d93e5cc0338"} Jul 10 17:39:37.249404 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:37.249148 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:37.249688 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:37.249558 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:37.516411 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:37.516327 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" event={"ID":"a57b3765-cc4f-407c-b815-f1c5958e66e8","Type":"ContainerStarted","Data":"a29ba9f64e6bf5817b6bb46de512d292caecc3f07daabe93fd00934c402275a0"} Jul 10 17:39:37.533855 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:37.533752 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4sr5" podStartSLOduration=4.746410017 podStartE2EDuration="26.533737043s" totalImagesPullingTime="21.787327026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:39:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:37.533268691 +0000 UTC m=+26.807595386" watchObservedRunningTime="2026-07-10 17:39:37.533737043 +0000 UTC m=+26.808063788" Jul 10 17:39:38.249074 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:38.249040 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:38.249232 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:38.249087 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:38.249232 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:38.249165 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:38.249338 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:38.249302 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:39.248532 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:39.248496 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:39.249081 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:39.248638 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:39.524178 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:39.523740 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntwdg" event={"ID":"63f4fcc3-2ddc-40fa-8a79-7f02580ce489","Type":"ContainerStarted","Data":"290b5ea66af35d15524c32e15b22f6b4cce700dfd90ee7b309b50e50489361e6"} Jul 10 17:39:39.528463 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:39.528373 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" event={"ID":"ac5b01b9-68ea-422b-a5a4-48b6a59d48df","Type":"ContainerStarted","Data":"5d392441912e611ee278a5dfd752d87a601a208b31c16470a47d6e5796eb5edc"} Jul 10 17:39:39.528787 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:39.528713 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:39.528920 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:39.528797 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:39.528920 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:39.528850 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:39.546950 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:39.546924 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:39.547170 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:39.547147 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:39:39.573942 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:39.573749 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" podStartSLOduration=9.455944881 podStartE2EDuration="28.573732719s" totalImagesPullingTime="19.117787838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:39.573681977 +0000 UTC m=+28.848008676" watchObservedRunningTime="2026-07-10 17:39:39.573732719 +0000 UTC m=+28.848059414" Jul 10 17:39:40.248616 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:40.248575 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:40.249200 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:40.248575 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:40.249200 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:40.248681 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:40.249200 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:40.248768 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:40.532407 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:40.532322 2556 generic.go:356] "Generic (PLEG): container finished" podID="63f4fcc3-2ddc-40fa-8a79-7f02580ce489" containerID="290b5ea66af35d15524c32e15b22f6b4cce700dfd90ee7b309b50e50489361e6" exitCode=0 Jul 10 17:39:40.532571 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:40.532407 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntwdg" event={"ID":"63f4fcc3-2ddc-40fa-8a79-7f02580ce489","Type":"ContainerDied","Data":"290b5ea66af35d15524c32e15b22f6b4cce700dfd90ee7b309b50e50489361e6"} Jul 10 17:39:41.250650 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:41.250611 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:41.251216 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:41.250739 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:41.536858 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:41.536813 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntwdg" event={"ID":"63f4fcc3-2ddc-40fa-8a79-7f02580ce489","Type":"ContainerStarted","Data":"3b45c995ffc1255af19bac61c7be5559c648e232adea52fe9a64a2d44e43e172"} Jul 10 17:39:41.685096 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:41.685058 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p58j7"] Jul 10 17:39:41.685276 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:41.685192 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:41.685334 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:41.685316 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:41.688199 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:41.688167 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wnspm"] Jul 10 17:39:41.688327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:41.688267 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:41.688389 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:41.688368 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:41.688784 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:41.688759 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t9t5m"] Jul 10 17:39:41.688867 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:41.688835 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:41.688931 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:41.688913 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:42.540601 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:42.540564 2556 generic.go:356] "Generic (PLEG): container finished" podID="63f4fcc3-2ddc-40fa-8a79-7f02580ce489" containerID="3b45c995ffc1255af19bac61c7be5559c648e232adea52fe9a64a2d44e43e172" exitCode=0 Jul 10 17:39:42.540997 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:42.540624 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntwdg" event={"ID":"63f4fcc3-2ddc-40fa-8a79-7f02580ce489","Type":"ContainerDied","Data":"3b45c995ffc1255af19bac61c7be5559c648e232adea52fe9a64a2d44e43e172"} Jul 10 17:39:43.248812 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:43.248773 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:43.249031 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:43.248791 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:43.249031 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:43.248898 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:43.249031 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:43.248902 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:43.249031 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:43.249001 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:43.249191 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:43.249052 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:44.548361 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:44.548327 2556 generic.go:356] "Generic (PLEG): container finished" podID="63f4fcc3-2ddc-40fa-8a79-7f02580ce489" containerID="af741dc18bbd3a7105c8cc127abd103e090bcf0d05ded81239bb835ec07cd504" exitCode=0 Jul 10 17:39:44.548806 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:44.548373 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntwdg" event={"ID":"63f4fcc3-2ddc-40fa-8a79-7f02580ce489","Type":"ContainerDied","Data":"af741dc18bbd3a7105c8cc127abd103e090bcf0d05ded81239bb835ec07cd504"} Jul 10 17:39:44.825118 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:44.825021 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:44.825287 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:44.825161 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:44.825287 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:44.825267 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret podName:779679cf-3637-4afa-a580-767d44eb5e09 nodeName:}" failed. No retries permitted until 2026-07-10 17:40:16.825238968 +0000 UTC m=+66.099565658 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret") pod "global-pull-secret-syncer-wnspm" (UID: "779679cf-3637-4afa-a580-767d44eb5e09") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:44.926350 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:44.926315 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:44.926527 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:44.926475 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:44.926575 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:44.926539 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs podName:17b57439-069e-42d9-b59f-976e14670388 nodeName:}" failed. No retries permitted until 2026-07-10 17:40:16.926523319 +0000 UTC m=+66.200850009 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs") pod "network-metrics-daemon-p58j7" (UID: "17b57439-069e-42d9-b59f-976e14670388") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:45.027711 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:45.027675 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntfx\" (UniqueName: \"kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx\") pod \"network-check-target-t9t5m\" (UID: \"cb66a47f-88eb-4ec2-a133-b8c5c85eed78\") " pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:45.028022 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:45.027882 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:39:45.028022 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:45.027911 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:39:45.028022 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:45.027926 2556 projected.go:196] Error preparing data for projected volume kube-api-access-bntfx for pod openshift-network-diagnostics/network-check-target-t9t5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:45.028022 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:45.028011 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx podName:cb66a47f-88eb-4ec2-a133-b8c5c85eed78 nodeName:}" failed. No retries permitted until 2026-07-10 17:40:17.027994873 +0000 UTC m=+66.302321550 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bntfx" (UniqueName: "kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx") pod "network-check-target-t9t5m" (UID: "cb66a47f-88eb-4ec2-a133-b8c5c85eed78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:45.249322 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:45.249283 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:45.249322 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:45.249320 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:45.249563 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:45.249282 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:45.249563 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:45.249444 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:39:45.249638 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:45.249569 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnspm" podUID="779679cf-3637-4afa-a580-767d44eb5e09" Jul 10 17:39:45.249671 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:45.249649 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9t5m" podUID="cb66a47f-88eb-4ec2-a133-b8c5c85eed78" Jul 10 17:39:46.027235 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.027199 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-103.ec2.internal" event="NodeReady" Jul 10 17:39:46.027824 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.027360 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 17:39:46.067702 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.067034 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8dblb"] Jul 10 17:39:46.069363 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.069337 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:46.069861 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.069834 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x7gfp"] Jul 10 17:39:46.071583 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.071564 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7gfp" Jul 10 17:39:46.071689 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.071630 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 17:39:46.071804 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.071787 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 17:39:46.072085 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.071957 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dbkj9\"" Jul 10 17:39:46.073831 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.073768 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 17:39:46.074093 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.074066 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h8cns\"" Jul 10 17:39:46.074226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.074210 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 17:39:46.074379 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.074366 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 17:39:46.080621 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.080600 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7gfp"] Jul 10 17:39:46.084916 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.083512 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8dblb"] Jul 10 17:39:46.135631 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.135591 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv9df\" (UniqueName: \"kubernetes.io/projected/1f6a4ab1-f10d-4111-812a-90d3f35f3484-kube-api-access-pv9df\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:46.135824 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.135638 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9035ea2d-129f-462f-8581-bfaf86b87718-cert\") pod \"ingress-canary-x7gfp\" (UID: \"9035ea2d-129f-462f-8581-bfaf86b87718\") " pod="openshift-ingress-canary/ingress-canary-x7gfp" Jul 10 17:39:46.135824 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.135668 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6a4ab1-f10d-4111-812a-90d3f35f3484-tmp-dir\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:46.135824 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.135720 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg9rb\" (UniqueName: \"kubernetes.io/projected/9035ea2d-129f-462f-8581-bfaf86b87718-kube-api-access-wg9rb\") pod \"ingress-canary-x7gfp\" (UID: \"9035ea2d-129f-462f-8581-bfaf86b87718\") " pod="openshift-ingress-canary/ingress-canary-x7gfp" Jul 10 17:39:46.135824 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.135767 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6a4ab1-f10d-4111-812a-90d3f35f3484-config-volume\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:46.135824 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.135808 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:46.236929 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.236897 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg9rb\" (UniqueName: \"kubernetes.io/projected/9035ea2d-129f-462f-8581-bfaf86b87718-kube-api-access-wg9rb\") pod \"ingress-canary-x7gfp\" (UID: \"9035ea2d-129f-462f-8581-bfaf86b87718\") " pod="openshift-ingress-canary/ingress-canary-x7gfp" Jul 10 17:39:46.237302 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.236960 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6a4ab1-f10d-4111-812a-90d3f35f3484-config-volume\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:46.237302 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.237019 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:46.237302 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.237063 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv9df\" (UniqueName: \"kubernetes.io/projected/1f6a4ab1-f10d-4111-812a-90d3f35f3484-kube-api-access-pv9df\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:46.237302 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.237093 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9035ea2d-129f-462f-8581-bfaf86b87718-cert\") pod \"ingress-canary-x7gfp\" (UID: \"9035ea2d-129f-462f-8581-bfaf86b87718\") " pod="openshift-ingress-canary/ingress-canary-x7gfp" Jul 10 17:39:46.237302 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.237114 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6a4ab1-f10d-4111-812a-90d3f35f3484-tmp-dir\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:46.237302 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:46.237189 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:39:46.237302 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:46.237263 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls podName:1f6a4ab1-f10d-4111-812a-90d3f35f3484 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:46.737239171 +0000 UTC m=+36.011565846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls") pod "dns-default-8dblb" (UID: "1f6a4ab1-f10d-4111-812a-90d3f35f3484") : secret "dns-default-metrics-tls" not found Jul 10 17:39:46.237608 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.237558 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6a4ab1-f10d-4111-812a-90d3f35f3484-tmp-dir\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:46.237752 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.237735 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6a4ab1-f10d-4111-812a-90d3f35f3484-config-volume\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:46.242551 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.242530 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9035ea2d-129f-462f-8581-bfaf86b87718-cert\") pod \"ingress-canary-x7gfp\" (UID: \"9035ea2d-129f-462f-8581-bfaf86b87718\") " pod="openshift-ingress-canary/ingress-canary-x7gfp" Jul 10 17:39:46.245868 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.245843 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv9df\" (UniqueName: \"kubernetes.io/projected/1f6a4ab1-f10d-4111-812a-90d3f35f3484-kube-api-access-pv9df\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:46.245984 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.245936 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg9rb\" (UniqueName: \"kubernetes.io/projected/9035ea2d-129f-462f-8581-bfaf86b87718-kube-api-access-wg9rb\") pod \"ingress-canary-x7gfp\" (UID: \"9035ea2d-129f-462f-8581-bfaf86b87718\") " pod="openshift-ingress-canary/ingress-canary-x7gfp" Jul 10 17:39:46.389481 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.389409 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7gfp" Jul 10 17:39:46.541264 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.541049 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7gfp"] Jul 10 17:39:46.545915 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:46.545877 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9035ea2d_129f_462f_8581_bfaf86b87718.slice/crio-0012e376c86f34528aefea47b0283450043a59fa1df6c9ff6c414e7e4a8973c9 WatchSource:0}: Error finding container 0012e376c86f34528aefea47b0283450043a59fa1df6c9ff6c414e7e4a8973c9: Status 404 returned error can't find the container with id 0012e376c86f34528aefea47b0283450043a59fa1df6c9ff6c414e7e4a8973c9 Jul 10 17:39:46.553484 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.553455 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7gfp" event={"ID":"9035ea2d-129f-462f-8581-bfaf86b87718","Type":"ContainerStarted","Data":"0012e376c86f34528aefea47b0283450043a59fa1df6c9ff6c414e7e4a8973c9"} Jul 10 17:39:46.740721 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:46.740547 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:46.740721 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:46.740669 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:39:46.740912 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:46.740739 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls podName:1f6a4ab1-f10d-4111-812a-90d3f35f3484 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:47.740723147 +0000 UTC m=+37.015049825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls") pod "dns-default-8dblb" (UID: "1f6a4ab1-f10d-4111-812a-90d3f35f3484") : secret "dns-default-metrics-tls" not found Jul 10 17:39:47.248504 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:47.248466 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:39:47.249207 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:47.248466 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:39:47.249207 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:47.249065 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:39:47.251453 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:47.251429 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 17:39:47.252568 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:47.252546 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 17:39:47.252684 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:47.252601 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 17:39:47.252749 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:47.252546 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n82vp\"" Jul 10 17:39:47.252749 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:47.252738 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 17:39:47.253032 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:47.253014 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rxjk8\"" Jul 10 17:39:47.751699 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:47.751661 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:47.751868 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:47.751829 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:39:47.751931 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:47.751912 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls podName:1f6a4ab1-f10d-4111-812a-90d3f35f3484 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:49.751889844 +0000 UTC m=+39.026216517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls") pod "dns-default-8dblb" (UID: "1f6a4ab1-f10d-4111-812a-90d3f35f3484") : secret "dns-default-metrics-tls" not found Jul 10 17:39:49.564214 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:49.564174 2556 generic.go:356] "Generic (PLEG): container finished" podID="63f4fcc3-2ddc-40fa-8a79-7f02580ce489" containerID="45af904f1e2572f6d87d101cc2959b2f11f0d93a89b92ba0358b82fed0816120" exitCode=0 Jul 10 17:39:49.564706 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:49.564258 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntwdg" event={"ID":"63f4fcc3-2ddc-40fa-8a79-7f02580ce489","Type":"ContainerDied","Data":"45af904f1e2572f6d87d101cc2959b2f11f0d93a89b92ba0358b82fed0816120"} Jul 10 17:39:49.766555 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:49.766514 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:49.766714 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:49.766659 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:39:49.766753 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:49.766727 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls podName:1f6a4ab1-f10d-4111-812a-90d3f35f3484 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:53.766709407 +0000 UTC m=+43.041036081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls") pod "dns-default-8dblb" (UID: "1f6a4ab1-f10d-4111-812a-90d3f35f3484") : secret "dns-default-metrics-tls" not found Jul 10 17:39:50.570591 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:50.570551 2556 generic.go:356] "Generic (PLEG): container finished" podID="63f4fcc3-2ddc-40fa-8a79-7f02580ce489" containerID="6ae25c9b5c714ceb1e86ba4ec97d4ca0e976751d51d173c443e1bb7dd94d94af" exitCode=0 Jul 10 17:39:50.571008 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:50.570625 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntwdg" event={"ID":"63f4fcc3-2ddc-40fa-8a79-7f02580ce489","Type":"ContainerDied","Data":"6ae25c9b5c714ceb1e86ba4ec97d4ca0e976751d51d173c443e1bb7dd94d94af"} Jul 10 17:39:51.575128 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:51.575007 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7gfp" event={"ID":"9035ea2d-129f-462f-8581-bfaf86b87718","Type":"ContainerStarted","Data":"5283bef781d83f857ac8ee8971d67e69be530325021e8b7b817aa561b84f8999"} Jul 10 17:39:51.578298 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:51.578271 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntwdg" event={"ID":"63f4fcc3-2ddc-40fa-8a79-7f02580ce489","Type":"ContainerStarted","Data":"cf288a82bb8b9f197cdcee4599b0d9cdf78603e3c7e11657d45f4cc2b2da0573"} Jul 10 17:39:51.590367 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:51.590316 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x7gfp" podStartSLOduration=0.941458721 podStartE2EDuration="5.590299864s" totalImagesPullingTime="4.648841143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:51.589326743 +0000 UTC m=+40.863653438" watchObservedRunningTime="2026-07-10 17:39:51.590299864 +0000 UTC m=+40.864626549" Jul 10 17:39:51.611165 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:51.611101 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ntwdg" podStartSLOduration=10.709983311 podStartE2EDuration="40.611078069s" totalImagesPullingTime="29.901094758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:39:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:51.609674552 +0000 UTC m=+40.884001248" watchObservedRunningTime="2026-07-10 17:39:51.611078069 +0000 UTC m=+40.885404767" Jul 10 17:39:53.796585 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:53.796544 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:39:53.797003 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:53.796704 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:39:53.797003 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:39:53.796771 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls podName:1f6a4ab1-f10d-4111-812a-90d3f35f3484 nodeName:}" failed. No retries permitted until 2026-07-10 17:40:01.796753768 +0000 UTC m=+51.071080446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls") pod "dns-default-8dblb" (UID: "1f6a4ab1-f10d-4111-812a-90d3f35f3484") : secret "dns-default-metrics-tls" not found Jul 10 17:39:59.147709 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.147676 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6"] Jul 10 17:39:59.181706 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.181674 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6"] Jul 10 17:39:59.181869 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.181777 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6" Jul 10 17:39:59.184695 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.184656 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 17:39:59.184848 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.184794 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 17:39:59.184905 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.184879 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 17:39:59.185944 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.185919 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-z5s2m\"" Jul 10 17:39:59.185944 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.185942 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 10 17:39:59.333750 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.333708 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vx7n\" (UniqueName: \"kubernetes.io/projected/23662b97-891f-4286-9a6c-8bf32ea70412-kube-api-access-9vx7n\") pod \"managed-serviceaccount-addon-agent-cf44d5d96-wlbd6\" (UID: \"23662b97-891f-4286-9a6c-8bf32ea70412\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6" Jul 10 17:39:59.333922 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.333765 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23662b97-891f-4286-9a6c-8bf32ea70412-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cf44d5d96-wlbd6\" (UID: \"23662b97-891f-4286-9a6c-8bf32ea70412\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6" Jul 10 17:39:59.435183 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.435095 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vx7n\" (UniqueName: \"kubernetes.io/projected/23662b97-891f-4286-9a6c-8bf32ea70412-kube-api-access-9vx7n\") pod \"managed-serviceaccount-addon-agent-cf44d5d96-wlbd6\" (UID: \"23662b97-891f-4286-9a6c-8bf32ea70412\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6" Jul 10 17:39:59.435183 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.435143 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23662b97-891f-4286-9a6c-8bf32ea70412-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cf44d5d96-wlbd6\" (UID: \"23662b97-891f-4286-9a6c-8bf32ea70412\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6" Jul 10 17:39:59.439556 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.439532 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23662b97-891f-4286-9a6c-8bf32ea70412-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cf44d5d96-wlbd6\" (UID: \"23662b97-891f-4286-9a6c-8bf32ea70412\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6" Jul 10 17:39:59.442825 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.442803 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vx7n\" (UniqueName: \"kubernetes.io/projected/23662b97-891f-4286-9a6c-8bf32ea70412-kube-api-access-9vx7n\") pod \"managed-serviceaccount-addon-agent-cf44d5d96-wlbd6\" (UID: \"23662b97-891f-4286-9a6c-8bf32ea70412\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6" Jul 10 17:39:59.502822 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.502786 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6" Jul 10 17:39:59.634728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:39:59.634688 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6"] Jul 10 17:39:59.638302 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:39:59.638270 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23662b97_891f_4286_9a6c_8bf32ea70412.slice/crio-2490f82cd01a2ac115c85e053dd7256ece1432dc84427456ddaa010a24dbd54d WatchSource:0}: Error finding container 2490f82cd01a2ac115c85e053dd7256ece1432dc84427456ddaa010a24dbd54d: Status 404 returned error can't find the container with id 2490f82cd01a2ac115c85e053dd7256ece1432dc84427456ddaa010a24dbd54d Jul 10 17:40:00.603904 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:00.603860 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6" event={"ID":"23662b97-891f-4286-9a6c-8bf32ea70412","Type":"ContainerStarted","Data":"2490f82cd01a2ac115c85e053dd7256ece1432dc84427456ddaa010a24dbd54d"} Jul 10 17:40:01.853623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:01.853574 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:40:01.854034 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:40:01.853755 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:40:01.854034 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:40:01.853839 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls podName:1f6a4ab1-f10d-4111-812a-90d3f35f3484 nodeName:}" failed. No retries permitted until 2026-07-10 17:40:17.853820608 +0000 UTC m=+67.128147284 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls") pod "dns-default-8dblb" (UID: "1f6a4ab1-f10d-4111-812a-90d3f35f3484") : secret "dns-default-metrics-tls" not found Jul 10 17:40:03.613611 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:03.613552 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6" event={"ID":"23662b97-891f-4286-9a6c-8bf32ea70412","Type":"ContainerStarted","Data":"c3f226f2ffdf197e77f41a502552cf7911af5008aaf2d2e6d8d0883d3bee2890"} Jul 10 17:40:03.627849 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:03.627655 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6" podStartSLOduration=1.422931049 podStartE2EDuration="4.627636938s" totalImagesPullingTime="3.204705889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:40:03.627070775 +0000 UTC m=+52.901397470" watchObservedRunningTime="2026-07-10 17:40:03.627636938 +0000 UTC m=+52.901963635" Jul 10 17:40:11.553237 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:11.553200 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9rhq" Jul 10 17:40:16.864311 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:16.864250 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:40:16.866931 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:16.866911 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 17:40:16.877222 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:16.877196 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/779679cf-3637-4afa-a580-767d44eb5e09-original-pull-secret\") pod \"global-pull-secret-syncer-wnspm\" (UID: \"779679cf-3637-4afa-a580-767d44eb5e09\") " pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:40:16.961341 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:16.961291 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnspm" Jul 10 17:40:16.965143 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:16.965120 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:40:16.967726 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:16.967706 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 17:40:16.975783 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:40:16.975745 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 17:40:16.975894 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:40:16.975837 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs podName:17b57439-069e-42d9-b59f-976e14670388 nodeName:}" failed. No retries permitted until 2026-07-10 17:41:20.97581298 +0000 UTC m=+130.250139656 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs") pod "network-metrics-daemon-p58j7" (UID: "17b57439-069e-42d9-b59f-976e14670388") : secret "metrics-daemon-secret" not found Jul 10 17:40:17.065620 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:17.065589 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntfx\" (UniqueName: \"kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx\") pod \"network-check-target-t9t5m\" (UID: \"cb66a47f-88eb-4ec2-a133-b8c5c85eed78\") " pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:40:17.068398 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:17.068374 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 17:40:17.078623 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:17.078599 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 17:40:17.087359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:17.087332 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wnspm"] Jul 10 17:40:17.089984 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:17.089939 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bntfx\" (UniqueName: \"kubernetes.io/projected/cb66a47f-88eb-4ec2-a133-b8c5c85eed78-kube-api-access-bntfx\") pod \"network-check-target-t9t5m\" (UID: \"cb66a47f-88eb-4ec2-a133-b8c5c85eed78\") " pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:40:17.090716 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:40:17.090694 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779679cf_3637_4afa_a580_767d44eb5e09.slice/crio-a7488546a69ab3d593ae740d30ccf6848f436c128278abdf7945c18a5c95e43c WatchSource:0}: Error finding container a7488546a69ab3d593ae740d30ccf6848f436c128278abdf7945c18a5c95e43c: Status 404 returned error can't find the container with id a7488546a69ab3d593ae740d30ccf6848f436c128278abdf7945c18a5c95e43c Jul 10 17:40:17.271516 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:17.271429 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n82vp\"" Jul 10 17:40:17.279775 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:17.279749 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:40:17.415422 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:17.415391 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t9t5m"] Jul 10 17:40:17.418392 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:40:17.418363 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb66a47f_88eb_4ec2_a133_b8c5c85eed78.slice/crio-177e7e15a9592be38c3553e445324192ca2ee4155736a10f9e06214142fa4c6d WatchSource:0}: Error finding container 177e7e15a9592be38c3553e445324192ca2ee4155736a10f9e06214142fa4c6d: Status 404 returned error can't find the container with id 177e7e15a9592be38c3553e445324192ca2ee4155736a10f9e06214142fa4c6d Jul 10 17:40:17.647093 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:17.647063 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wnspm" event={"ID":"779679cf-3637-4afa-a580-767d44eb5e09","Type":"ContainerStarted","Data":"a7488546a69ab3d593ae740d30ccf6848f436c128278abdf7945c18a5c95e43c"} Jul 10 17:40:17.648089 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:17.648064 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t9t5m" event={"ID":"cb66a47f-88eb-4ec2-a133-b8c5c85eed78","Type":"ContainerStarted","Data":"177e7e15a9592be38c3553e445324192ca2ee4155736a10f9e06214142fa4c6d"} Jul 10 17:40:17.873828 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:17.873792 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:40:17.874212 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:40:17.873912 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:40:17.874212 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:40:17.873991 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls podName:1f6a4ab1-f10d-4111-812a-90d3f35f3484 nodeName:}" failed. No retries permitted until 2026-07-10 17:40:49.8739575 +0000 UTC m=+99.148284175 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls") pod "dns-default-8dblb" (UID: "1f6a4ab1-f10d-4111-812a-90d3f35f3484") : secret "dns-default-metrics-tls" not found Jul 10 17:40:21.660698 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:21.660595 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t9t5m" event={"ID":"cb66a47f-88eb-4ec2-a133-b8c5c85eed78","Type":"ContainerStarted","Data":"e9475049d6f126e428f90c0f2f8b33a2b0a847f33d8c8d121294a4d983573c92"} Jul 10 17:40:21.660698 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:21.660671 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:40:21.675910 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:21.675848 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t9t5m" podStartSLOduration=67.487518132 podStartE2EDuration="1m10.675828016s" totalImagesPullingTime="3.188309884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:40:21.674456005 +0000 UTC m=+70.948782708" watchObservedRunningTime="2026-07-10 17:40:21.675828016 +0000 UTC m=+70.950154710" Jul 10 17:40:22.664877 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:22.664836 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wnspm" event={"ID":"779679cf-3637-4afa-a580-767d44eb5e09","Type":"ContainerStarted","Data":"f2c435313e09982b08bbd1802b5dcc6f16d0d27b50d62b30e86dcd7476049085"} Jul 10 17:40:22.678664 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:22.678616 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wnspm" podStartSLOduration=66.618902922 podStartE2EDuration="1m11.678601508s" totalImagesPullingTime="5.059698586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:40:22.678137601 +0000 UTC m=+71.952464297" watchObservedRunningTime="2026-07-10 17:40:22.678601508 +0000 UTC m=+71.952928202" Jul 10 17:40:49.892601 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:49.892462 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:40:49.892601 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:40:49.892598 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:40:49.893133 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:40:49.892668 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls podName:1f6a4ab1-f10d-4111-812a-90d3f35f3484 nodeName:}" failed. No retries permitted until 2026-07-10 17:41:53.892652325 +0000 UTC m=+163.166978998 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls") pod "dns-default-8dblb" (UID: "1f6a4ab1-f10d-4111-812a-90d3f35f3484") : secret "dns-default-metrics-tls" not found Jul 10 17:40:52.667752 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:40:52.667718 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t9t5m" Jul 10 17:41:21.005955 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:41:21.005918 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:41:21.006446 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:41:21.006069 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 17:41:21.006446 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:41:21.006136 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs podName:17b57439-069e-42d9-b59f-976e14670388 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:23.006119966 +0000 UTC m=+252.280446639 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs") pod "network-metrics-daemon-p58j7" (UID: "17b57439-069e-42d9-b59f-976e14670388") : secret "metrics-daemon-secret" not found Jul 10 17:41:49.082498 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:41:49.082433 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8dblb" podUID="1f6a4ab1-f10d-4111-812a-90d3f35f3484" Jul 10 17:41:49.876708 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:41:49.876679 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8dblb" Jul 10 17:41:50.276848 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:41:50.276759 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-p58j7" podUID="17b57439-069e-42d9-b59f-976e14670388" Jul 10 17:41:53.932715 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:41:53.932668 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:41:53.933237 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:41:53.932830 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:41:53.933237 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:41:53.932912 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls podName:1f6a4ab1-f10d-4111-812a-90d3f35f3484 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:55.932895006 +0000 UTC m=+285.207221679 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls") pod "dns-default-8dblb" (UID: "1f6a4ab1-f10d-4111-812a-90d3f35f3484") : secret "dns-default-metrics-tls" not found Jul 10 17:42:03.912049 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:03.912012 2556 generic.go:356] "Generic (PLEG): container finished" podID="23662b97-891f-4286-9a6c-8bf32ea70412" containerID="c3f226f2ffdf197e77f41a502552cf7911af5008aaf2d2e6d8d0883d3bee2890" exitCode=255 Jul 10 17:42:03.912411 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:03.912073 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6" event={"ID":"23662b97-891f-4286-9a6c-8bf32ea70412","Type":"ContainerDied","Data":"c3f226f2ffdf197e77f41a502552cf7911af5008aaf2d2e6d8d0883d3bee2890"} Jul 10 17:42:03.917721 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:03.917699 2556 scope.go:117] "RemoveContainer" containerID="c3f226f2ffdf197e77f41a502552cf7911af5008aaf2d2e6d8d0883d3bee2890" Jul 10 17:42:04.916144 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:04.916110 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf44d5d96-wlbd6" event={"ID":"23662b97-891f-4286-9a6c-8bf32ea70412","Type":"ContainerStarted","Data":"e91524b1b598482efad9c4a1461ef9b2031335d26d8281a1feaf5cd5819bc8ed"} Jul 10 17:42:05.248837 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:05.248738 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:42:59.675631 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.675589 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h"] Jul 10 17:42:59.677846 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.677823 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-xcxtg"] Jul 10 17:42:59.678004 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.677929 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:42:59.679551 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.679531 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-jqdnm"] Jul 10 17:42:59.679674 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.679645 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:42:59.680443 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.680420 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 10 17:42:59.681586 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.681568 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jqdnm" Jul 10 17:42:59.681720 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.681703 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2wdg2\"" Jul 10 17:42:59.682138 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.681928 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 17:42:59.682138 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.681955 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 10 17:42:59.682138 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.682021 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x7lwk\"" Jul 10 17:42:59.682138 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.681994 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 17:42:59.682138 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.682054 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 10 17:42:59.682138 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.681985 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 10 17:42:59.682138 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.682130 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 10 17:42:59.682458 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.682320 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:42:59.684095 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.684077 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-lr94g\"" Jul 10 17:42:59.686618 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.686597 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-9xdxz"] Jul 10 17:42:59.690155 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.690132 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 10 17:42:59.691023 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.691001 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-9xdxz" Jul 10 17:42:59.691782 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.691762 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h"] Jul 10 17:42:59.693387 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.693368 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:42:59.693484 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.693452 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 10 17:42:59.693484 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.693468 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fkzwz\"" Jul 10 17:42:59.693601 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.693566 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-jqdnm"] Jul 10 17:42:59.694437 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.694415 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-xcxtg"] Jul 10 17:42:59.702889 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.702862 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-9xdxz"] Jul 10 17:42:59.759459 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.759421 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7"] Jul 10 17:42:59.761450 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.761426 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" Jul 10 17:42:59.763266 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.763242 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4"] Jul 10 17:42:59.763947 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.763912 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-crlzb\"" Jul 10 17:42:59.764063 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.764047 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 10 17:42:59.764276 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.764259 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:42:59.764356 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.764308 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 10 17:42:59.765351 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.765334 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" Jul 10 17:42:59.769442 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.769269 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:42:59.769442 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.769400 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 10 17:42:59.769442 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.769418 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 10 17:42:59.769655 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.769511 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 10 17:42:59.769802 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.769784 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vlvfn\"" Jul 10 17:42:59.773873 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.773849 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/76091f5f-5621-4255-ac27-2aca61b38b53-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:42:59.774017 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.773880 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jshmq\" (UniqueName: \"kubernetes.io/projected/76091f5f-5621-4255-ac27-2aca61b38b53-kube-api-access-jshmq\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:42:59.774017 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.773904 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvkx\" (UniqueName: \"kubernetes.io/projected/1539cdfd-1ef2-4cbb-86ef-9eaf178c19e3-kube-api-access-brvkx\") pod \"volume-data-source-validator-78495597f6-9xdxz\" (UID: \"1539cdfd-1ef2-4cbb-86ef-9eaf178c19e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-9xdxz" Jul 10 17:42:59.774134 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.774105 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/672f8194-f0a2-460c-891e-1c6e7ca21f81-config\") pod \"console-operator-7866497494-xcxtg\" (UID: \"672f8194-f0a2-460c-891e-1c6e7ca21f81\") " pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:42:59.774190 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.774163 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7"] Jul 10 17:42:59.774190 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.774141 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/672f8194-f0a2-460c-891e-1c6e7ca21f81-serving-cert\") pod \"console-operator-7866497494-xcxtg\" (UID: \"672f8194-f0a2-460c-891e-1c6e7ca21f81\") " pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:42:59.774298 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.774207 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:42:59.774298 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.774256 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/672f8194-f0a2-460c-891e-1c6e7ca21f81-trusted-ca\") pod \"console-operator-7866497494-xcxtg\" (UID: \"672f8194-f0a2-460c-891e-1c6e7ca21f81\") " pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:42:59.774298 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.774280 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp94k\" (UniqueName: \"kubernetes.io/projected/fa763257-aa6b-49f2-b222-8df3a534da29-kube-api-access-lp94k\") pod \"cluster-samples-operator-65db858894-nggx7\" (UID: \"fa763257-aa6b-49f2-b222-8df3a534da29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" Jul 10 17:42:59.774437 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.774301 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzsf6\" (UniqueName: \"kubernetes.io/projected/672f8194-f0a2-460c-891e-1c6e7ca21f81-kube-api-access-wzsf6\") pod \"console-operator-7866497494-xcxtg\" (UID: \"672f8194-f0a2-460c-891e-1c6e7ca21f81\") " pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:42:59.774437 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.774317 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc5sp\" (UniqueName: \"kubernetes.io/projected/0a4d3207-e503-4c54-b8fc-2ece3e74249d-kube-api-access-dc5sp\") pod \"network-check-source-869dcfcdcd-jqdnm\" (UID: \"0a4d3207-e503-4c54-b8fc-2ece3e74249d\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jqdnm" Jul 10 17:42:59.774437 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.774399 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-nggx7\" (UID: \"fa763257-aa6b-49f2-b222-8df3a534da29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" Jul 10 17:42:59.778003 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.777956 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4"] Jul 10 17:42:59.875097 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875055 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/672f8194-f0a2-460c-891e-1c6e7ca21f81-config\") pod \"console-operator-7866497494-xcxtg\" (UID: \"672f8194-f0a2-460c-891e-1c6e7ca21f81\") " pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:42:59.875097 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875092 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/672f8194-f0a2-460c-891e-1c6e7ca21f81-serving-cert\") pod \"console-operator-7866497494-xcxtg\" (UID: \"672f8194-f0a2-460c-891e-1c6e7ca21f81\") " pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:42:59.875376 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875117 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:42:59.875376 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875135 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/672f8194-f0a2-460c-891e-1c6e7ca21f81-trusted-ca\") pod \"console-operator-7866497494-xcxtg\" (UID: \"672f8194-f0a2-460c-891e-1c6e7ca21f81\") " pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:42:59.875376 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875160 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71a927f1-68cc-4c42-9b68-eed6282b657b-config\") pod \"service-ca-operator-7695fc6c8c-x5kg4\" (UID: \"71a927f1-68cc-4c42-9b68-eed6282b657b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" Jul 10 17:42:59.875376 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875183 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvj97\" (UniqueName: \"kubernetes.io/projected/71a927f1-68cc-4c42-9b68-eed6282b657b-kube-api-access-dvj97\") pod \"service-ca-operator-7695fc6c8c-x5kg4\" (UID: \"71a927f1-68cc-4c42-9b68-eed6282b657b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" Jul 10 17:42:59.875376 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875210 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp94k\" (UniqueName: \"kubernetes.io/projected/fa763257-aa6b-49f2-b222-8df3a534da29-kube-api-access-lp94k\") pod \"cluster-samples-operator-65db858894-nggx7\" (UID: \"fa763257-aa6b-49f2-b222-8df3a534da29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" Jul 10 17:42:59.875376 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:42:59.875227 2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 17:42:59.875376 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875233 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71a927f1-68cc-4c42-9b68-eed6282b657b-serving-cert\") pod \"service-ca-operator-7695fc6c8c-x5kg4\" (UID: \"71a927f1-68cc-4c42-9b68-eed6282b657b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" Jul 10 17:42:59.875376 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:42:59.875363 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls podName:76091f5f-5621-4255-ac27-2aca61b38b53 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:00.375332339 +0000 UTC m=+229.649659020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-q8w4h" (UID: "76091f5f-5621-4255-ac27-2aca61b38b53") : secret "cluster-monitoring-operator-tls" not found Jul 10 17:42:59.875822 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875408 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzsf6\" (UniqueName: \"kubernetes.io/projected/672f8194-f0a2-460c-891e-1c6e7ca21f81-kube-api-access-wzsf6\") pod \"console-operator-7866497494-xcxtg\" (UID: \"672f8194-f0a2-460c-891e-1c6e7ca21f81\") " pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:42:59.875822 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875439 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc5sp\" (UniqueName: \"kubernetes.io/projected/0a4d3207-e503-4c54-b8fc-2ece3e74249d-kube-api-access-dc5sp\") pod \"network-check-source-869dcfcdcd-jqdnm\" (UID: \"0a4d3207-e503-4c54-b8fc-2ece3e74249d\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jqdnm" Jul 10 17:42:59.875822 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875508 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-nggx7\" (UID: \"fa763257-aa6b-49f2-b222-8df3a534da29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" Jul 10 17:42:59.875822 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875552 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/76091f5f-5621-4255-ac27-2aca61b38b53-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:42:59.875822 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875595 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jshmq\" (UniqueName: \"kubernetes.io/projected/76091f5f-5621-4255-ac27-2aca61b38b53-kube-api-access-jshmq\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:42:59.875822 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875622 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvkx\" (UniqueName: \"kubernetes.io/projected/1539cdfd-1ef2-4cbb-86ef-9eaf178c19e3-kube-api-access-brvkx\") pod \"volume-data-source-validator-78495597f6-9xdxz\" (UID: \"1539cdfd-1ef2-4cbb-86ef-9eaf178c19e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-9xdxz" Jul 10 17:42:59.876089 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:42:59.875861 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 17:42:59.876089 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.875906 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/672f8194-f0a2-460c-891e-1c6e7ca21f81-config\") pod \"console-operator-7866497494-xcxtg\" (UID: \"672f8194-f0a2-460c-891e-1c6e7ca21f81\") " pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:42:59.876089 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.876072 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/672f8194-f0a2-460c-891e-1c6e7ca21f81-trusted-ca\") pod \"console-operator-7866497494-xcxtg\" (UID: \"672f8194-f0a2-460c-891e-1c6e7ca21f81\") " pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:42:59.876189 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:42:59.876107 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls podName:fa763257-aa6b-49f2-b222-8df3a534da29 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:00.376065566 +0000 UTC m=+229.650392268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls") pod "cluster-samples-operator-65db858894-nggx7" (UID: "fa763257-aa6b-49f2-b222-8df3a534da29") : secret "samples-operator-tls" not found Jul 10 17:42:59.876270 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.876252 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/76091f5f-5621-4255-ac27-2aca61b38b53-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:42:59.877652 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.877637 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/672f8194-f0a2-460c-891e-1c6e7ca21f81-serving-cert\") pod \"console-operator-7866497494-xcxtg\" (UID: \"672f8194-f0a2-460c-891e-1c6e7ca21f81\") " pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:42:59.886872 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.886843 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvkx\" (UniqueName: \"kubernetes.io/projected/1539cdfd-1ef2-4cbb-86ef-9eaf178c19e3-kube-api-access-brvkx\") pod \"volume-data-source-validator-78495597f6-9xdxz\" (UID: \"1539cdfd-1ef2-4cbb-86ef-9eaf178c19e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-9xdxz" Jul 10 17:42:59.887021 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.886849 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp94k\" (UniqueName: \"kubernetes.io/projected/fa763257-aa6b-49f2-b222-8df3a534da29-kube-api-access-lp94k\") pod \"cluster-samples-operator-65db858894-nggx7\" (UID: \"fa763257-aa6b-49f2-b222-8df3a534da29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" Jul 10 17:42:59.887224 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.887204 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzsf6\" (UniqueName: \"kubernetes.io/projected/672f8194-f0a2-460c-891e-1c6e7ca21f81-kube-api-access-wzsf6\") pod \"console-operator-7866497494-xcxtg\" (UID: \"672f8194-f0a2-460c-891e-1c6e7ca21f81\") " pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:42:59.887530 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.887509 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jshmq\" (UniqueName: \"kubernetes.io/projected/76091f5f-5621-4255-ac27-2aca61b38b53-kube-api-access-jshmq\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:42:59.887573 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.887559 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc5sp\" (UniqueName: \"kubernetes.io/projected/0a4d3207-e503-4c54-b8fc-2ece3e74249d-kube-api-access-dc5sp\") pod \"network-check-source-869dcfcdcd-jqdnm\" (UID: \"0a4d3207-e503-4c54-b8fc-2ece3e74249d\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jqdnm" Jul 10 17:42:59.976439 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.976347 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71a927f1-68cc-4c42-9b68-eed6282b657b-config\") pod \"service-ca-operator-7695fc6c8c-x5kg4\" (UID: \"71a927f1-68cc-4c42-9b68-eed6282b657b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" Jul 10 17:42:59.976439 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.976383 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvj97\" (UniqueName: \"kubernetes.io/projected/71a927f1-68cc-4c42-9b68-eed6282b657b-kube-api-access-dvj97\") pod \"service-ca-operator-7695fc6c8c-x5kg4\" (UID: \"71a927f1-68cc-4c42-9b68-eed6282b657b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" Jul 10 17:42:59.976439 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.976402 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71a927f1-68cc-4c42-9b68-eed6282b657b-serving-cert\") pod \"service-ca-operator-7695fc6c8c-x5kg4\" (UID: \"71a927f1-68cc-4c42-9b68-eed6282b657b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" Jul 10 17:42:59.977021 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.976995 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71a927f1-68cc-4c42-9b68-eed6282b657b-config\") pod \"service-ca-operator-7695fc6c8c-x5kg4\" (UID: \"71a927f1-68cc-4c42-9b68-eed6282b657b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" Jul 10 17:42:59.978698 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.978680 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71a927f1-68cc-4c42-9b68-eed6282b657b-serving-cert\") pod \"service-ca-operator-7695fc6c8c-x5kg4\" (UID: \"71a927f1-68cc-4c42-9b68-eed6282b657b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" Jul 10 17:42:59.984554 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:42:59.984526 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvj97\" (UniqueName: \"kubernetes.io/projected/71a927f1-68cc-4c42-9b68-eed6282b657b-kube-api-access-dvj97\") pod \"service-ca-operator-7695fc6c8c-x5kg4\" (UID: \"71a927f1-68cc-4c42-9b68-eed6282b657b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" Jul 10 17:43:00.000553 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:00.000510 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:43:00.007424 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:00.007402 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jqdnm" Jul 10 17:43:00.012319 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:00.012296 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-9xdxz" Jul 10 17:43:00.077912 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:00.077861 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" Jul 10 17:43:00.148996 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:00.148844 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-xcxtg"] Jul 10 17:43:00.152784 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:00.152747 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672f8194_f0a2_460c_891e_1c6e7ca21f81.slice/crio-d6d44af5468861fa14d3aebb346e15df383f3e03bab4d2dbd9807e609ff59d8c WatchSource:0}: Error finding container d6d44af5468861fa14d3aebb346e15df383f3e03bab4d2dbd9807e609ff59d8c: Status 404 returned error can't find the container with id d6d44af5468861fa14d3aebb346e15df383f3e03bab4d2dbd9807e609ff59d8c Jul 10 17:43:00.226539 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:00.226453 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4"] Jul 10 17:43:00.230523 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:00.230498 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a927f1_68cc_4c42_9b68_eed6282b657b.slice/crio-6d5fd293a12cd67191b1e29ac6ce36b4844c559a107312b4bb6ed760dca226cc WatchSource:0}: Error finding container 6d5fd293a12cd67191b1e29ac6ce36b4844c559a107312b4bb6ed760dca226cc: Status 404 returned error can't find the container with id 6d5fd293a12cd67191b1e29ac6ce36b4844c559a107312b4bb6ed760dca226cc Jul 10 17:43:00.368833 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:00.368797 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-9xdxz"] Jul 10 17:43:00.371566 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:00.371528 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-jqdnm"] Jul 10 17:43:00.371947 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:00.371923 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1539cdfd_1ef2_4cbb_86ef_9eaf178c19e3.slice/crio-65aad02e34f4995c09093b72d5c6d7a17f969d2c41d4bc78ae2d37279d2686d1 WatchSource:0}: Error finding container 65aad02e34f4995c09093b72d5c6d7a17f969d2c41d4bc78ae2d37279d2686d1: Status 404 returned error can't find the container with id 65aad02e34f4995c09093b72d5c6d7a17f969d2c41d4bc78ae2d37279d2686d1 Jul 10 17:43:00.374465 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:00.374438 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4d3207_e503_4c54_b8fc_2ece3e74249d.slice/crio-d96b26693041bc17240bd4c27f8bcc6f37eda6bda9de5d86ac4ce8c961ec045d WatchSource:0}: Error finding container d96b26693041bc17240bd4c27f8bcc6f37eda6bda9de5d86ac4ce8c961ec045d: Status 404 returned error can't find the container with id d96b26693041bc17240bd4c27f8bcc6f37eda6bda9de5d86ac4ce8c961ec045d Jul 10 17:43:00.380747 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:00.380723 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:43:00.380845 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:00.380783 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-nggx7\" (UID: \"fa763257-aa6b-49f2-b222-8df3a534da29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" Jul 10 17:43:00.380905 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:00.380866 2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 17:43:00.380905 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:00.380877 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 17:43:00.381023 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:00.380929 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls podName:fa763257-aa6b-49f2-b222-8df3a534da29 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:01.380911441 +0000 UTC m=+230.655238116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls") pod "cluster-samples-operator-65db858894-nggx7" (UID: "fa763257-aa6b-49f2-b222-8df3a534da29") : secret "samples-operator-tls" not found Jul 10 17:43:00.381023 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:00.380943 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls podName:76091f5f-5621-4255-ac27-2aca61b38b53 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:01.380936577 +0000 UTC m=+230.655263251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-q8w4h" (UID: "76091f5f-5621-4255-ac27-2aca61b38b53") : secret "cluster-monitoring-operator-tls" not found Jul 10 17:43:01.054312 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:01.054272 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" event={"ID":"71a927f1-68cc-4c42-9b68-eed6282b657b","Type":"ContainerStarted","Data":"6d5fd293a12cd67191b1e29ac6ce36b4844c559a107312b4bb6ed760dca226cc"} Jul 10 17:43:01.056520 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:01.056490 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jqdnm" event={"ID":"0a4d3207-e503-4c54-b8fc-2ece3e74249d","Type":"ContainerStarted","Data":"57cb1aebbaf27999a70d481da53ec07c635a054d00c511ea88ebe933c353a098"} Jul 10 17:43:01.056613 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:01.056545 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jqdnm" event={"ID":"0a4d3207-e503-4c54-b8fc-2ece3e74249d","Type":"ContainerStarted","Data":"d96b26693041bc17240bd4c27f8bcc6f37eda6bda9de5d86ac4ce8c961ec045d"} Jul 10 17:43:01.058493 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:01.058428 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-9xdxz" event={"ID":"1539cdfd-1ef2-4cbb-86ef-9eaf178c19e3","Type":"ContainerStarted","Data":"65aad02e34f4995c09093b72d5c6d7a17f969d2c41d4bc78ae2d37279d2686d1"} Jul 10 17:43:01.060488 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:01.060438 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-xcxtg" event={"ID":"672f8194-f0a2-460c-891e-1c6e7ca21f81","Type":"ContainerStarted","Data":"d6d44af5468861fa14d3aebb346e15df383f3e03bab4d2dbd9807e609ff59d8c"} Jul 10 17:43:01.388489 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:01.388446 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-nggx7\" (UID: \"fa763257-aa6b-49f2-b222-8df3a534da29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" Jul 10 17:43:01.388676 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:01.388549 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:43:01.388676 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:01.388635 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 17:43:01.388676 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:01.388674 2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 17:43:01.388844 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:01.388721 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls podName:fa763257-aa6b-49f2-b222-8df3a534da29 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:03.38869875 +0000 UTC m=+232.663025440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls") pod "cluster-samples-operator-65db858894-nggx7" (UID: "fa763257-aa6b-49f2-b222-8df3a534da29") : secret "samples-operator-tls" not found Jul 10 17:43:01.388844 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:01.388741 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls podName:76091f5f-5621-4255-ac27-2aca61b38b53 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:03.388731312 +0000 UTC m=+232.663057999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-q8w4h" (UID: "76091f5f-5621-4255-ac27-2aca61b38b53") : secret "cluster-monitoring-operator-tls" not found Jul 10 17:43:03.068711 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.068629 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/0.log" Jul 10 17:43:03.068711 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.068669 2556 generic.go:356] "Generic (PLEG): container finished" podID="672f8194-f0a2-460c-891e-1c6e7ca21f81" containerID="b19b5b5f433838af270c58e8cbe5f107f5d3ea8770945bd47136854468f332db" exitCode=255 Jul 10 17:43:03.069228 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.068746 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-xcxtg" event={"ID":"672f8194-f0a2-460c-891e-1c6e7ca21f81","Type":"ContainerDied","Data":"b19b5b5f433838af270c58e8cbe5f107f5d3ea8770945bd47136854468f332db"} Jul 10 17:43:03.069228 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.069032 2556 scope.go:117] "RemoveContainer" containerID="b19b5b5f433838af270c58e8cbe5f107f5d3ea8770945bd47136854468f332db" Jul 10 17:43:03.070430 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.070405 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" event={"ID":"71a927f1-68cc-4c42-9b68-eed6282b657b","Type":"ContainerStarted","Data":"7afa507232d8500ca9742658e4051959dfa6a628fbe5fb7cc1054de261b6b19a"} Jul 10 17:43:03.072012 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.071985 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-9xdxz" event={"ID":"1539cdfd-1ef2-4cbb-86ef-9eaf178c19e3","Type":"ContainerStarted","Data":"73811f14b71db5b6ddd6d1de350c3a56f4176b35e90b7c51ddc6ee06473df63f"} Jul 10 17:43:03.086439 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.086388 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jqdnm" podStartSLOduration=4.086369416 podStartE2EDuration="4.086369416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:01.072605643 +0000 UTC m=+230.346932339" watchObservedRunningTime="2026-07-10 17:43:03.086369416 +0000 UTC m=+232.360696112" Jul 10 17:43:03.099147 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.099102 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-9xdxz" podStartSLOduration=1.711685232 podStartE2EDuration="4.099085916s" totalImagesPullingTime="2.387400684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:03.098699622 +0000 UTC m=+232.373026328" watchObservedRunningTime="2026-07-10 17:43:03.099085916 +0000 UTC m=+232.373412646" Jul 10 17:43:03.115003 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.114927 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" podStartSLOduration=1.580254882 podStartE2EDuration="4.11491002s" totalImagesPullingTime="2.534655138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:03.114664757 +0000 UTC m=+232.388991455" watchObservedRunningTime="2026-07-10 17:43:03.11491002 +0000 UTC m=+232.389236715" Jul 10 17:43:03.404057 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.404016 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-nggx7\" (UID: \"fa763257-aa6b-49f2-b222-8df3a534da29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" Jul 10 17:43:03.404235 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.404100 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:43:03.404235 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:03.404184 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 17:43:03.404235 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:03.404215 2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 17:43:03.404350 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:03.404260 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls podName:fa763257-aa6b-49f2-b222-8df3a534da29 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:07.404238915 +0000 UTC m=+236.678565600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls") pod "cluster-samples-operator-65db858894-nggx7" (UID: "fa763257-aa6b-49f2-b222-8df3a534da29") : secret "samples-operator-tls" not found Jul 10 17:43:03.404350 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:03.404275 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls podName:76091f5f-5621-4255-ac27-2aca61b38b53 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:07.404268956 +0000 UTC m=+236.678595628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-q8w4h" (UID: "76091f5f-5621-4255-ac27-2aca61b38b53") : secret "cluster-monitoring-operator-tls" not found Jul 10 17:43:03.528318 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.528289 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hs657"] Jul 10 17:43:03.531431 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.531415 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:03.533852 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.533826 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 17:43:03.533852 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.533827 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 17:43:03.534076 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.533857 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 17:43:03.534076 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.533865 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 17:43:03.534076 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.533904 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-66sjw\"" Jul 10 17:43:03.541785 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.541762 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hs657"] Jul 10 17:43:03.605244 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.605205 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7cl4\" (UniqueName: \"kubernetes.io/projected/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-api-access-q7cl4\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:03.605244 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.605246 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:03.605502 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.605322 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d1f472c-aae2-436c-944e-754bafcadbdf-crio-socket\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:03.605502 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.605358 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:03.605502 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.605405 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d1f472c-aae2-436c-944e-754bafcadbdf-data-volume\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:03.705930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.705824 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7cl4\" (UniqueName: \"kubernetes.io/projected/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-api-access-q7cl4\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:03.705930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.705881 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:03.706208 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.706007 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d1f472c-aae2-436c-944e-754bafcadbdf-crio-socket\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:03.706208 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:03.706023 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:43:03.706208 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.706043 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:03.706208 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:03.706095 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls podName:3d1f472c-aae2-436c-944e-754bafcadbdf nodeName:}" failed. No retries permitted until 2026-07-10 17:43:04.206075487 +0000 UTC m=+233.480402173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hs657" (UID: "3d1f472c-aae2-436c-944e-754bafcadbdf") : secret "insights-runtime-extractor-tls" not found Jul 10 17:43:03.706208 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:03.706115 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:43:03.706208 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.706127 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d1f472c-aae2-436c-944e-754bafcadbdf-crio-socket\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:03.706208 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.706131 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d1f472c-aae2-436c-944e-754bafcadbdf-data-volume\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:03.706208 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:03.706166 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm podName:3d1f472c-aae2-436c-944e-754bafcadbdf nodeName:}" failed. No retries permitted until 2026-07-10 17:43:04.206151416 +0000 UTC m=+233.480478094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hs657" (UID: "3d1f472c-aae2-436c-944e-754bafcadbdf") : configmap "kube-rbac-proxy" not found Jul 10 17:43:03.706552 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.706442 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d1f472c-aae2-436c-944e-754bafcadbdf-data-volume\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:03.714636 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:03.714607 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7cl4\" (UniqueName: \"kubernetes.io/projected/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-api-access-q7cl4\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:04.077280 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.077200 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 17:43:04.077706 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.077641 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/0.log" Jul 10 17:43:04.077706 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.077683 2556 generic.go:356] "Generic (PLEG): container finished" podID="672f8194-f0a2-460c-891e-1c6e7ca21f81" containerID="eb78cff05fa8ab678f62e27ff276452dd6a2362415849e370522c02d4f49369e" exitCode=255 Jul 10 17:43:04.077852 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.077823 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-xcxtg" event={"ID":"672f8194-f0a2-460c-891e-1c6e7ca21f81","Type":"ContainerDied","Data":"eb78cff05fa8ab678f62e27ff276452dd6a2362415849e370522c02d4f49369e"} Jul 10 17:43:04.077906 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.077896 2556 scope.go:117] "RemoveContainer" containerID="b19b5b5f433838af270c58e8cbe5f107f5d3ea8770945bd47136854468f332db" Jul 10 17:43:04.078222 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.078198 2556 scope.go:117] "RemoveContainer" containerID="eb78cff05fa8ab678f62e27ff276452dd6a2362415849e370522c02d4f49369e" Jul 10 17:43:04.078404 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:04.078383 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-xcxtg_openshift-console-operator(672f8194-f0a2-460c-891e-1c6e7ca21f81)\"" pod="openshift-console-operator/console-operator-7866497494-xcxtg" podUID="672f8194-f0a2-460c-891e-1c6e7ca21f81" Jul 10 17:43:04.210257 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.210215 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:04.210452 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.210276 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:04.210452 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:04.210366 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:43:04.210452 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:04.210430 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls podName:3d1f472c-aae2-436c-944e-754bafcadbdf nodeName:}" failed. No retries permitted until 2026-07-10 17:43:05.210414238 +0000 UTC m=+234.484740912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hs657" (UID: "3d1f472c-aae2-436c-944e-754bafcadbdf") : secret "insights-runtime-extractor-tls" not found Jul 10 17:43:04.210614 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:04.210370 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:43:04.210614 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:04.210541 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm podName:3d1f472c-aae2-436c-944e-754bafcadbdf nodeName:}" failed. No retries permitted until 2026-07-10 17:43:05.210521624 +0000 UTC m=+234.484848317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hs657" (UID: "3d1f472c-aae2-436c-944e-754bafcadbdf") : configmap "kube-rbac-proxy" not found Jul 10 17:43:04.428470 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.428428 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pdjp8"] Jul 10 17:43:04.432702 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.432679 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pdjp8" Jul 10 17:43:04.435050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.435029 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 17:43:04.435179 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.435058 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qzskc\"" Jul 10 17:43:04.435179 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.435090 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 17:43:04.441480 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.441448 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pdjp8"] Jul 10 17:43:04.513686 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.513646 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvv7h\" (UniqueName: \"kubernetes.io/projected/fa889ec6-c374-4a61-9465-061926984d83-kube-api-access-dvv7h\") pod \"migrator-7f59b54f6c-pdjp8\" (UID: \"fa889ec6-c374-4a61-9465-061926984d83\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pdjp8" Jul 10 17:43:04.614875 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.614833 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvv7h\" (UniqueName: \"kubernetes.io/projected/fa889ec6-c374-4a61-9465-061926984d83-kube-api-access-dvv7h\") pod \"migrator-7f59b54f6c-pdjp8\" (UID: \"fa889ec6-c374-4a61-9465-061926984d83\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pdjp8" Jul 10 17:43:04.622349 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.622315 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvv7h\" (UniqueName: \"kubernetes.io/projected/fa889ec6-c374-4a61-9465-061926984d83-kube-api-access-dvv7h\") pod \"migrator-7f59b54f6c-pdjp8\" (UID: \"fa889ec6-c374-4a61-9465-061926984d83\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pdjp8" Jul 10 17:43:04.742650 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.742567 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pdjp8" Jul 10 17:43:04.868885 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:04.868854 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pdjp8"] Jul 10 17:43:04.872287 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:04.872259 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa889ec6_c374_4a61_9465_061926984d83.slice/crio-7ddc6eb44ba896dab20292823adbb1a7ae7a1ca69761614357fe86f4aff4ee4e WatchSource:0}: Error finding container 7ddc6eb44ba896dab20292823adbb1a7ae7a1ca69761614357fe86f4aff4ee4e: Status 404 returned error can't find the container with id 7ddc6eb44ba896dab20292823adbb1a7ae7a1ca69761614357fe86f4aff4ee4e Jul 10 17:43:05.082526 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:05.082451 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 17:43:05.082922 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:05.082822 2556 scope.go:117] "RemoveContainer" containerID="eb78cff05fa8ab678f62e27ff276452dd6a2362415849e370522c02d4f49369e" Jul 10 17:43:05.083076 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:05.083004 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-xcxtg_openshift-console-operator(672f8194-f0a2-460c-891e-1c6e7ca21f81)\"" pod="openshift-console-operator/console-operator-7866497494-xcxtg" podUID="672f8194-f0a2-460c-891e-1c6e7ca21f81" Jul 10 17:43:05.083632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:05.083614 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pdjp8" event={"ID":"fa889ec6-c374-4a61-9465-061926984d83","Type":"ContainerStarted","Data":"7ddc6eb44ba896dab20292823adbb1a7ae7a1ca69761614357fe86f4aff4ee4e"} Jul 10 17:43:05.218918 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:05.218868 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:05.222821 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:05.219300 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:05.222821 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:05.219416 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:43:05.222821 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:05.219504 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls podName:3d1f472c-aae2-436c-944e-754bafcadbdf nodeName:}" failed. No retries permitted until 2026-07-10 17:43:07.219483559 +0000 UTC m=+236.493810246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hs657" (UID: "3d1f472c-aae2-436c-944e-754bafcadbdf") : secret "insights-runtime-extractor-tls" not found Jul 10 17:43:05.222821 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:05.219547 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:43:05.222821 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:05.219610 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm podName:3d1f472c-aae2-436c-944e-754bafcadbdf nodeName:}" failed. No retries permitted until 2026-07-10 17:43:07.219593405 +0000 UTC m=+236.493920098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hs657" (UID: "3d1f472c-aae2-436c-944e-754bafcadbdf") : configmap "kube-rbac-proxy" not found Jul 10 17:43:05.510475 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:05.510450 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-flzpz_3a04a38b-bb00-4ab0-b1d2-433b32e982e2/dns-node-resolver/0.log" Jul 10 17:43:06.090945 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.090920 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pdjp8" event={"ID":"fa889ec6-c374-4a61-9465-061926984d83","Type":"ContainerStarted","Data":"8da6a1e989a53c355f462c107b59b61c7a773f746f5cdccef203b75db533a9cf"} Jul 10 17:43:06.310562 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.310489 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4sdqt_67a35b3c-e9c2-45f7-9518-9754c6015eb0/node-ca/0.log" Jul 10 17:43:06.583570 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.583489 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-j8vd9"] Jul 10 17:43:06.586501 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.586485 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" Jul 10 17:43:06.589073 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.589048 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 17:43:06.589073 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.589060 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 17:43:06.590000 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.589962 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 17:43:06.590121 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.589980 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ddqhs\"" Jul 10 17:43:06.590121 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.589983 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 17:43:06.593440 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.593416 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-j8vd9"] Jul 10 17:43:06.630082 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.630039 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d309a6c8-7b14-4d4c-906f-2b7aa456314d-signing-key\") pod \"service-ca-79f68b9f4f-j8vd9\" (UID: \"d309a6c8-7b14-4d4c-906f-2b7aa456314d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" Jul 10 17:43:06.630082 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.630087 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qd2j\" (UniqueName: \"kubernetes.io/projected/d309a6c8-7b14-4d4c-906f-2b7aa456314d-kube-api-access-6qd2j\") pod \"service-ca-79f68b9f4f-j8vd9\" (UID: \"d309a6c8-7b14-4d4c-906f-2b7aa456314d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" Jul 10 17:43:06.630313 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.630133 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d309a6c8-7b14-4d4c-906f-2b7aa456314d-signing-cabundle\") pod \"service-ca-79f68b9f4f-j8vd9\" (UID: \"d309a6c8-7b14-4d4c-906f-2b7aa456314d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" Jul 10 17:43:06.730518 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.730481 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d309a6c8-7b14-4d4c-906f-2b7aa456314d-signing-key\") pod \"service-ca-79f68b9f4f-j8vd9\" (UID: \"d309a6c8-7b14-4d4c-906f-2b7aa456314d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" Jul 10 17:43:06.730518 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.730518 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qd2j\" (UniqueName: \"kubernetes.io/projected/d309a6c8-7b14-4d4c-906f-2b7aa456314d-kube-api-access-6qd2j\") pod \"service-ca-79f68b9f4f-j8vd9\" (UID: \"d309a6c8-7b14-4d4c-906f-2b7aa456314d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" Jul 10 17:43:06.730779 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.730576 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d309a6c8-7b14-4d4c-906f-2b7aa456314d-signing-cabundle\") pod \"service-ca-79f68b9f4f-j8vd9\" (UID: \"d309a6c8-7b14-4d4c-906f-2b7aa456314d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" Jul 10 17:43:06.731288 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.731263 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d309a6c8-7b14-4d4c-906f-2b7aa456314d-signing-cabundle\") pod \"service-ca-79f68b9f4f-j8vd9\" (UID: \"d309a6c8-7b14-4d4c-906f-2b7aa456314d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" Jul 10 17:43:06.733049 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.733028 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d309a6c8-7b14-4d4c-906f-2b7aa456314d-signing-key\") pod \"service-ca-79f68b9f4f-j8vd9\" (UID: \"d309a6c8-7b14-4d4c-906f-2b7aa456314d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" Jul 10 17:43:06.740872 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.740849 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qd2j\" (UniqueName: \"kubernetes.io/projected/d309a6c8-7b14-4d4c-906f-2b7aa456314d-kube-api-access-6qd2j\") pod \"service-ca-79f68b9f4f-j8vd9\" (UID: \"d309a6c8-7b14-4d4c-906f-2b7aa456314d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" Jul 10 17:43:06.896154 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:06.896117 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" Jul 10 17:43:07.020458 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:07.020421 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-j8vd9"] Jul 10 17:43:07.023287 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:07.023263 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd309a6c8_7b14_4d4c_906f_2b7aa456314d.slice/crio-55611344f844ba6fde05e3cf39d2008f50eec55ece76600c43b4226e73bc882e WatchSource:0}: Error finding container 55611344f844ba6fde05e3cf39d2008f50eec55ece76600c43b4226e73bc882e: Status 404 returned error can't find the container with id 55611344f844ba6fde05e3cf39d2008f50eec55ece76600c43b4226e73bc882e Jul 10 17:43:07.097001 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:07.096947 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" event={"ID":"d309a6c8-7b14-4d4c-906f-2b7aa456314d","Type":"ContainerStarted","Data":"20893f884030b6881644aaa188f36e889b68fd1adbb0b1d9d604307d56b790e0"} Jul 10 17:43:07.097384 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:07.097029 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" event={"ID":"d309a6c8-7b14-4d4c-906f-2b7aa456314d","Type":"ContainerStarted","Data":"55611344f844ba6fde05e3cf39d2008f50eec55ece76600c43b4226e73bc882e"} Jul 10 17:43:07.098839 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:07.098813 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pdjp8" event={"ID":"fa889ec6-c374-4a61-9465-061926984d83","Type":"ContainerStarted","Data":"8d63de895ce52bbc0a11657a0bcfc451c7562b830f2c666b4d2763c88a456f1c"} Jul 10 17:43:07.118521 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:07.118459 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-j8vd9" podStartSLOduration=1.1184397289999999 podStartE2EDuration="1.118439729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:07.115796066 +0000 UTC m=+236.390122760" watchObservedRunningTime="2026-07-10 17:43:07.118439729 +0000 UTC m=+236.392766425" Jul 10 17:43:07.135164 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:07.135116 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pdjp8" podStartSLOduration=1.9858518250000001 podStartE2EDuration="3.135095927s" totalImagesPullingTime="1.149244102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:07.134046007 +0000 UTC m=+236.408372703" watchObservedRunningTime="2026-07-10 17:43:07.135095927 +0000 UTC m=+236.409422621" Jul 10 17:43:07.234619 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:07.234531 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:07.234619 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:07.234581 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:07.234839 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:07.234682 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:43:07.234839 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:07.234741 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:43:07.234839 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:07.234749 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls podName:3d1f472c-aae2-436c-944e-754bafcadbdf nodeName:}" failed. No retries permitted until 2026-07-10 17:43:11.234734051 +0000 UTC m=+240.509060726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hs657" (UID: "3d1f472c-aae2-436c-944e-754bafcadbdf") : secret "insights-runtime-extractor-tls" not found Jul 10 17:43:07.234839 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:07.234802 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm podName:3d1f472c-aae2-436c-944e-754bafcadbdf nodeName:}" failed. No retries permitted until 2026-07-10 17:43:11.234784844 +0000 UTC m=+240.509111516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hs657" (UID: "3d1f472c-aae2-436c-944e-754bafcadbdf") : configmap "kube-rbac-proxy" not found Jul 10 17:43:07.436050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:07.436005 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-nggx7\" (UID: \"fa763257-aa6b-49f2-b222-8df3a534da29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" Jul 10 17:43:07.436255 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:07.436119 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:43:07.436255 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:07.436156 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 17:43:07.436255 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:07.436215 2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 17:43:07.436255 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:07.436218 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls podName:fa763257-aa6b-49f2-b222-8df3a534da29 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:15.436203114 +0000 UTC m=+244.710529786 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls") pod "cluster-samples-operator-65db858894-nggx7" (UID: "fa763257-aa6b-49f2-b222-8df3a534da29") : secret "samples-operator-tls" not found Jul 10 17:43:07.436443 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:07.436274 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls podName:76091f5f-5621-4255-ac27-2aca61b38b53 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:15.436256032 +0000 UTC m=+244.710582715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-q8w4h" (UID: "76091f5f-5621-4255-ac27-2aca61b38b53") : secret "cluster-monitoring-operator-tls" not found Jul 10 17:43:07.511846 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:07.511761 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x7gfp_9035ea2d-129f-462f-8581-bfaf86b87718/serve-healthcheck-canary/0.log" Jul 10 17:43:10.001474 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:10.001368 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:43:10.001474 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:10.001437 2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:43:10.001953 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:10.001923 2556 scope.go:117] "RemoveContainer" containerID="eb78cff05fa8ab678f62e27ff276452dd6a2362415849e370522c02d4f49369e" Jul 10 17:43:10.002130 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:10.002109 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-xcxtg_openshift-console-operator(672f8194-f0a2-460c-891e-1c6e7ca21f81)\"" pod="openshift-console-operator/console-operator-7866497494-xcxtg" podUID="672f8194-f0a2-460c-891e-1c6e7ca21f81" Jul 10 17:43:11.269100 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:11.269066 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:11.269522 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:11.269125 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:11.269522 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:11.269202 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:43:11.269522 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:11.269218 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:43:11.269522 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:11.269268 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm podName:3d1f472c-aae2-436c-944e-754bafcadbdf nodeName:}" failed. No retries permitted until 2026-07-10 17:43:19.269249824 +0000 UTC m=+248.543576497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hs657" (UID: "3d1f472c-aae2-436c-944e-754bafcadbdf") : configmap "kube-rbac-proxy" not found Jul 10 17:43:11.269522 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:11.269281 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls podName:3d1f472c-aae2-436c-944e-754bafcadbdf nodeName:}" failed. No retries permitted until 2026-07-10 17:43:19.269275826 +0000 UTC m=+248.543602498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hs657" (UID: "3d1f472c-aae2-436c-944e-754bafcadbdf") : secret "insights-runtime-extractor-tls" not found Jul 10 17:43:15.504311 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:15.504271 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:43:15.504711 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:15.504326 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-nggx7\" (UID: \"fa763257-aa6b-49f2-b222-8df3a534da29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" Jul 10 17:43:15.506901 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:15.506874 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa763257-aa6b-49f2-b222-8df3a534da29-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-nggx7\" (UID: \"fa763257-aa6b-49f2-b222-8df3a534da29\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" Jul 10 17:43:15.507032 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:15.507005 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76091f5f-5621-4255-ac27-2aca61b38b53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-q8w4h\" (UID: \"76091f5f-5621-4255-ac27-2aca61b38b53\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:43:15.596564 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:15.596530 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2wdg2\"" Jul 10 17:43:15.604544 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:15.604517 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" Jul 10 17:43:15.673598 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:15.673574 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-crlzb\"" Jul 10 17:43:15.681732 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:15.681709 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" Jul 10 17:43:15.737238 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:15.737201 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h"] Jul 10 17:43:15.740641 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:15.740602 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76091f5f_5621_4255_ac27_2aca61b38b53.slice/crio-b79a8d3f338d301a2478d64d3277a5ded2d71563bf286ee6d327e94d2a9a9143 WatchSource:0}: Error finding container b79a8d3f338d301a2478d64d3277a5ded2d71563bf286ee6d327e94d2a9a9143: Status 404 returned error can't find the container with id b79a8d3f338d301a2478d64d3277a5ded2d71563bf286ee6d327e94d2a9a9143 Jul 10 17:43:15.815186 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:15.815150 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7"] Jul 10 17:43:16.128573 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:16.128536 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" event={"ID":"fa763257-aa6b-49f2-b222-8df3a534da29","Type":"ContainerStarted","Data":"1f42cb00413f31741f45dda9199993a0b53614643d01cd289a2016cfcf0203f0"} Jul 10 17:43:16.129685 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:16.129664 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" event={"ID":"76091f5f-5621-4255-ac27-2aca61b38b53","Type":"ContainerStarted","Data":"b79a8d3f338d301a2478d64d3277a5ded2d71563bf286ee6d327e94d2a9a9143"} Jul 10 17:43:19.142040 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:19.141996 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" event={"ID":"76091f5f-5621-4255-ac27-2aca61b38b53","Type":"ContainerStarted","Data":"48ab9a80a9baffcc365c193392310f959b2131d77a8a4c733fd2512b5fe67c3a"} Jul 10 17:43:19.143654 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:19.143627 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" event={"ID":"fa763257-aa6b-49f2-b222-8df3a534da29","Type":"ContainerStarted","Data":"e1f3db01726f16d4f975bbfcfa17aa2268aea4dab21b737a172a08a24fd90413"} Jul 10 17:43:19.143750 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:19.143673 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" event={"ID":"fa763257-aa6b-49f2-b222-8df3a534da29","Type":"ContainerStarted","Data":"51dce3227ea7e12b865b59849d39ba063cae2d7c337a63bee58ef097e64b4522"} Jul 10 17:43:19.158870 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:19.158819 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-q8w4h" podStartSLOduration=17.738633975 podStartE2EDuration="20.158803941s" totalImagesPullingTime="2.420169966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:19.157924797 +0000 UTC m=+248.432251492" watchObservedRunningTime="2026-07-10 17:43:19.158803941 +0000 UTC m=+248.433130636" Jul 10 17:43:19.174333 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:19.174281 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-nggx7" podStartSLOduration=17.860767716 podStartE2EDuration="20.174262886s" totalImagesPullingTime="2.31349517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:19.173679437 +0000 UTC m=+248.448006132" watchObservedRunningTime="2026-07-10 17:43:19.174262886 +0000 UTC m=+248.448589608" Jul 10 17:43:19.336548 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:19.336505 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:19.336706 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:19.336594 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:19.337088 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:19.337060 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1f472c-aae2-436c-944e-754bafcadbdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:19.339057 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:19.339038 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1f472c-aae2-436c-944e-754bafcadbdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hs657\" (UID: \"3d1f472c-aae2-436c-944e-754bafcadbdf\") " pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:19.442944 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:19.442861 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-66sjw\"" Jul 10 17:43:19.450930 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:19.450908 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hs657" Jul 10 17:43:19.584182 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:19.584154 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hs657"] Jul 10 17:43:19.587147 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:19.587105 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1f472c_aae2_436c_944e_754bafcadbdf.slice/crio-a540e6d9345f807622ee2d7e5a1a27d5c73546a62ec470e18baf60b3e475f105 WatchSource:0}: Error finding container a540e6d9345f807622ee2d7e5a1a27d5c73546a62ec470e18baf60b3e475f105: Status 404 returned error can't find the container with id a540e6d9345f807622ee2d7e5a1a27d5c73546a62ec470e18baf60b3e475f105 Jul 10 17:43:20.150368 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:20.150326 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hs657" event={"ID":"3d1f472c-aae2-436c-944e-754bafcadbdf","Type":"ContainerStarted","Data":"a540e6d9345f807622ee2d7e5a1a27d5c73546a62ec470e18baf60b3e475f105"} Jul 10 17:43:20.249680 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:20.249646 2556 scope.go:117] "RemoveContainer" containerID="eb78cff05fa8ab678f62e27ff276452dd6a2362415849e370522c02d4f49369e" Jul 10 17:43:21.159221 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:21.159195 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 17:43:21.159654 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:21.159287 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-xcxtg" event={"ID":"672f8194-f0a2-460c-891e-1c6e7ca21f81","Type":"ContainerStarted","Data":"cd91b94167f97180d61352a8b2c3d96ad05b16148b2b8e6e8e5077a3b7c51855"} Jul 10 17:43:21.159654 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:21.159576 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:43:21.176084 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:21.176035 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-xcxtg" podStartSLOduration=19.565699825 podStartE2EDuration="22.17601752s" totalImagesPullingTime="2.610317695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:21.175181678 +0000 UTC m=+250.449508398" watchObservedRunningTime="2026-07-10 17:43:21.17601752 +0000 UTC m=+250.450344216" Jul 10 17:43:21.847465 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:21.847433 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-xcxtg" Jul 10 17:43:23.066130 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:23.066091 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:43:23.068581 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:23.068516 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17b57439-069e-42d9-b59f-976e14670388-metrics-certs\") pod \"network-metrics-daemon-p58j7\" (UID: \"17b57439-069e-42d9-b59f-976e14670388\") " pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:43:23.254413 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:23.254379 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rxjk8\"" Jul 10 17:43:23.260016 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:23.259959 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p58j7" Jul 10 17:43:23.519235 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:23.519206 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p58j7"] Jul 10 17:43:23.522596 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:23.522570 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b57439_069e_42d9_b59f_976e14670388.slice/crio-5af6c232a24d7db6609db14765d18fc3b415285b52e077d6ad9cc81038dca16e WatchSource:0}: Error finding container 5af6c232a24d7db6609db14765d18fc3b415285b52e077d6ad9cc81038dca16e: Status 404 returned error can't find the container with id 5af6c232a24d7db6609db14765d18fc3b415285b52e077d6ad9cc81038dca16e Jul 10 17:43:24.171273 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:24.171236 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hs657" event={"ID":"3d1f472c-aae2-436c-944e-754bafcadbdf","Type":"ContainerStarted","Data":"828ffd3615b8f9c137dff6aca8138a12d46f98c85a4f56b1aa445a074e6f643f"} Jul 10 17:43:24.172605 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:24.172575 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p58j7" event={"ID":"17b57439-069e-42d9-b59f-976e14670388","Type":"ContainerStarted","Data":"5af6c232a24d7db6609db14765d18fc3b415285b52e077d6ad9cc81038dca16e"} Jul 10 17:43:25.178840 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:25.178805 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hs657" event={"ID":"3d1f472c-aae2-436c-944e-754bafcadbdf","Type":"ContainerStarted","Data":"66a89db9379e8facba705ebd00e078798395bf315f35495acc96bd2a1eb9f775"} Jul 10 17:43:25.180948 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:25.180913 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p58j7" event={"ID":"17b57439-069e-42d9-b59f-976e14670388","Type":"ContainerStarted","Data":"fd3cca553188c6103f96d9a4aaeb6f831075dddbddddcd0e9d1c0bf9a23e358e"} Jul 10 17:43:25.181103 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:25.180994 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p58j7" event={"ID":"17b57439-069e-42d9-b59f-976e14670388","Type":"ContainerStarted","Data":"53f806ea282227aa9e3f638d67630404626ba9f35ea273c6083919f91411ae33"} Jul 10 17:43:25.199849 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:25.199789 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p58j7" podStartSLOduration=253.349160757 podStartE2EDuration="4m14.199765707s" totalImagesPullingTime="850.60495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:25.1986773 +0000 UTC m=+254.473003994" watchObservedRunningTime="2026-07-10 17:43:25.199765707 +0000 UTC m=+254.474092406" Jul 10 17:43:27.190327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:27.190239 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hs657" event={"ID":"3d1f472c-aae2-436c-944e-754bafcadbdf","Type":"ContainerStarted","Data":"df7a2d18169de5944c0177b7de6babfafd275fa87910670f53f91aafdae4e0e5"} Jul 10 17:43:27.208867 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:27.208814 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hs657" podStartSLOduration=17.107156163 podStartE2EDuration="24.208795976s" totalImagesPullingTime="7.101639813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:43:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:27.206761306 +0000 UTC m=+256.481088002" watchObservedRunningTime="2026-07-10 17:43:27.208795976 +0000 UTC m=+256.483122672" Jul 10 17:43:35.721266 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.721186 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66799fcc67-wsrrk"] Jul 10 17:43:35.723913 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.723890 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.727629 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.727600 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 17:43:35.727755 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.727679 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 17:43:35.727755 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.727694 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z6jhd\"" Jul 10 17:43:35.727755 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.727705 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 17:43:35.727755 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.727630 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 17:43:35.727755 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.727625 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 17:43:35.728037 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.727600 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 17:43:35.728037 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.727608 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 17:43:35.736485 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.736460 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66799fcc67-wsrrk"] Jul 10 17:43:35.736859 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.736838 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 17:43:35.868764 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.868724 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfnlz\" (UniqueName: \"kubernetes.io/projected/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-kube-api-access-sfnlz\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.868945 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.868805 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-trusted-ca-bundle\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.868945 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.868829 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-oauth-config\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.868945 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.868857 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-service-ca\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.868945 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.868912 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-serving-cert\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.869130 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.868949 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-oauth-serving-cert\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.869130 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.869054 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-config\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.969601 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.969557 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-trusted-ca-bundle\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.969601 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.969596 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-oauth-config\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.969865 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.969613 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-service-ca\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.969865 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.969637 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-serving-cert\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.969865 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.969651 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-oauth-serving-cert\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.969865 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.969803 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-config\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.970115 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.969870 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfnlz\" (UniqueName: \"kubernetes.io/projected/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-kube-api-access-sfnlz\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.970456 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.970398 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-service-ca\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.970565 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.970542 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-oauth-serving-cert\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.970685 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.970664 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-config\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.970902 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.970883 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-trusted-ca-bundle\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.972606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.972554 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-serving-cert\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.972719 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.972700 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-oauth-config\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:35.981039 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:35.978332 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfnlz\" (UniqueName: \"kubernetes.io/projected/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-kube-api-access-sfnlz\") pod \"console-66799fcc67-wsrrk\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:36.036565 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.036528 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:36.074779 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.074749 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl"] Jul 10 17:43:36.080310 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.080226 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:36.087466 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.085689 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w8nqr"] Jul 10 17:43:36.090039 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.089146 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 10 17:43:36.090039 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.089545 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 17:43:36.090039 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.090030 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 10 17:43:36.090349 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.090242 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9pznh\"" Jul 10 17:43:36.092729 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.092278 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl"] Jul 10 17:43:36.092729 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.092404 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.095372 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.095350 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 17:43:36.095739 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.095725 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lp7qm\"" Jul 10 17:43:36.095875 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.095852 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 17:43:36.096159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.096145 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 17:43:36.097754 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.096339 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-wrdnb"] Jul 10 17:43:36.105275 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.105253 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.113325 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.113304 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 10 17:43:36.116039 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.115791 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 10 17:43:36.116039 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.114527 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-wrdnb"] Jul 10 17:43:36.116039 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.115688 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5vgpg\"" Jul 10 17:43:36.116039 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.115595 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 10 17:43:36.174328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.172673 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-textfile\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.174328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.172741 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-wtmp\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.174328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.172804 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1b70818-5e06-41c7-88fc-a321618a3349-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-gt6gl\" (UID: \"f1b70818-5e06-41c7-88fc-a321618a3349\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:36.174328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.172886 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-accelerators-collector-config\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.174328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.172918 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61455718-aab9-4990-b93b-2c1d7be1532d-sys\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.174328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.172944 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmzxt\" (UniqueName: \"kubernetes.io/projected/61455718-aab9-4990-b93b-2c1d7be1532d-kube-api-access-dmzxt\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.174328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.172993 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1b70818-5e06-41c7-88fc-a321618a3349-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-gt6gl\" (UID: \"f1b70818-5e06-41c7-88fc-a321618a3349\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:36.174328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.173030 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-tls\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.174328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.173054 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61455718-aab9-4990-b93b-2c1d7be1532d-metrics-client-ca\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.174328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.173082 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61455718-aab9-4990-b93b-2c1d7be1532d-root\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.174328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.173109 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1b70818-5e06-41c7-88fc-a321618a3349-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-gt6gl\" (UID: \"f1b70818-5e06-41c7-88fc-a321618a3349\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:36.174328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.173173 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.174328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.173200 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8gd\" (UniqueName: \"kubernetes.io/projected/f1b70818-5e06-41c7-88fc-a321618a3349-kube-api-access-5j8gd\") pod \"openshift-state-metrics-84bd68b896-gt6gl\" (UID: \"f1b70818-5e06-41c7-88fc-a321618a3349\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:36.215182 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.215131 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66799fcc67-wsrrk"] Jul 10 17:43:36.218497 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:36.218340 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc0ca44_1c79_4fb6_abb5_15c92c8e9b74.slice/crio-55fe88a66d0cf68dd9e9672c39a573e31ef07f64edcdc2abe355979424a1fc59 WatchSource:0}: Error finding container 55fe88a66d0cf68dd9e9672c39a573e31ef07f64edcdc2abe355979424a1fc59: Status 404 returned error can't find the container with id 55fe88a66d0cf68dd9e9672c39a573e31ef07f64edcdc2abe355979424a1fc59 Jul 10 17:43:36.273947 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.273909 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.274165 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.273959 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8gd\" (UniqueName: \"kubernetes.io/projected/f1b70818-5e06-41c7-88fc-a321618a3349-kube-api-access-5j8gd\") pod \"openshift-state-metrics-84bd68b896-gt6gl\" (UID: \"f1b70818-5e06-41c7-88fc-a321618a3349\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:36.274165 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274015 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-textfile\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.274165 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274055 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-wtmp\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.274327 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274193 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-wtmp\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.274382 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274354 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.274443 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274400 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1b70818-5e06-41c7-88fc-a321618a3349-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-gt6gl\" (UID: \"f1b70818-5e06-41c7-88fc-a321618a3349\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:36.274500 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274439 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.274500 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274472 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274505 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-accelerators-collector-config\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:36.274515 2556 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274508 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-textfile\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:36.274579 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1b70818-5e06-41c7-88fc-a321618a3349-openshift-state-metrics-tls podName:f1b70818-5e06-41c7-88fc-a321618a3349 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:36.774558311 +0000 UTC m=+266.048884998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f1b70818-5e06-41c7-88fc-a321618a3349-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-gt6gl" (UID: "f1b70818-5e06-41c7-88fc-a321618a3349") : secret "openshift-state-metrics-tls" not found Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274630 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61455718-aab9-4990-b93b-2c1d7be1532d-sys\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274674 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274719 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmzxt\" (UniqueName: \"kubernetes.io/projected/61455718-aab9-4990-b93b-2c1d7be1532d-kube-api-access-dmzxt\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274742 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1b70818-5e06-41c7-88fc-a321618a3349-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-gt6gl\" (UID: \"f1b70818-5e06-41c7-88fc-a321618a3349\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274781 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rzhm\" (UniqueName: \"kubernetes.io/projected/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-kube-api-access-8rzhm\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274820 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-tls\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274842 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61455718-aab9-4990-b93b-2c1d7be1532d-metrics-client-ca\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274869 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61455718-aab9-4990-b93b-2c1d7be1532d-root\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274893 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1b70818-5e06-41c7-88fc-a321618a3349-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-gt6gl\" (UID: \"f1b70818-5e06-41c7-88fc-a321618a3349\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.274917 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.275130 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-accelerators-collector-config\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.282359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.275189 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61455718-aab9-4990-b93b-2c1d7be1532d-sys\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.283054 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.275231 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61455718-aab9-4990-b93b-2c1d7be1532d-root\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.283054 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.275631 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61455718-aab9-4990-b93b-2c1d7be1532d-metrics-client-ca\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.283054 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.275682 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1b70818-5e06-41c7-88fc-a321618a3349-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-gt6gl\" (UID: \"f1b70818-5e06-41c7-88fc-a321618a3349\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:36.283054 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:36.275730 2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 17:43:36.283054 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:36.275777 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-tls podName:61455718-aab9-4990-b93b-2c1d7be1532d nodeName:}" failed. No retries permitted until 2026-07-10 17:43:36.775761781 +0000 UTC m=+266.050088468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-tls") pod "node-exporter-w8nqr" (UID: "61455718-aab9-4990-b93b-2c1d7be1532d") : secret "node-exporter-tls" not found Jul 10 17:43:36.283054 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.277060 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.283054 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.277835 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1b70818-5e06-41c7-88fc-a321618a3349-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-gt6gl\" (UID: \"f1b70818-5e06-41c7-88fc-a321618a3349\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:36.285767 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.285739 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8gd\" (UniqueName: \"kubernetes.io/projected/f1b70818-5e06-41c7-88fc-a321618a3349-kube-api-access-5j8gd\") pod \"openshift-state-metrics-84bd68b896-gt6gl\" (UID: \"f1b70818-5e06-41c7-88fc-a321618a3349\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:36.286483 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.286453 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmzxt\" (UniqueName: \"kubernetes.io/projected/61455718-aab9-4990-b93b-2c1d7be1532d-kube-api-access-dmzxt\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.375479 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.375439 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.375663 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.375496 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.375663 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.375537 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.375663 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.375562 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.375663 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.375592 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rzhm\" (UniqueName: \"kubernetes.io/projected/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-kube-api-access-8rzhm\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.375961 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.375938 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.376424 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.376376 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.376424 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.376403 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.376424 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.376414 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.378178 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.378157 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.378498 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.378481 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.385014 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.384992 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rzhm\" (UniqueName: \"kubernetes.io/projected/6a5a8a6f-6afc-4114-998d-4102f3d4e91e-kube-api-access-8rzhm\") pod \"kube-state-metrics-785778ff74-wrdnb\" (UID: \"6a5a8a6f-6afc-4114-998d-4102f3d4e91e\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.429629 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.429585 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" Jul 10 17:43:36.582616 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.582585 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-wrdnb"] Jul 10 17:43:36.585573 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:36.585547 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5a8a6f_6afc_4114_998d_4102f3d4e91e.slice/crio-447b341c582db33032e4df2ba583eb07ea240451c363390b36457dec9bdee28a WatchSource:0}: Error finding container 447b341c582db33032e4df2ba583eb07ea240451c363390b36457dec9bdee28a: Status 404 returned error can't find the container with id 447b341c582db33032e4df2ba583eb07ea240451c363390b36457dec9bdee28a Jul 10 17:43:36.778407 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.778314 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-tls\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.778863 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.778412 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1b70818-5e06-41c7-88fc-a321618a3349-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-gt6gl\" (UID: \"f1b70818-5e06-41c7-88fc-a321618a3349\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:36.780891 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.780866 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61455718-aab9-4990-b93b-2c1d7be1532d-node-exporter-tls\") pod \"node-exporter-w8nqr\" (UID: \"61455718-aab9-4990-b93b-2c1d7be1532d\") " pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:36.781020 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:36.780875 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1b70818-5e06-41c7-88fc-a321618a3349-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-gt6gl\" (UID: \"f1b70818-5e06-41c7-88fc-a321618a3349\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:37.008824 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.008792 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" Jul 10 17:43:37.021172 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.021145 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w8nqr" Jul 10 17:43:37.053418 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:37.053378 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61455718_aab9_4990_b93b_2c1d7be1532d.slice/crio-9874a679c82ea1de958c8a2fc0958cd76ac79c5df6a6199b7b3345ab6c685ce3 WatchSource:0}: Error finding container 9874a679c82ea1de958c8a2fc0958cd76ac79c5df6a6199b7b3345ab6c685ce3: Status 404 returned error can't find the container with id 9874a679c82ea1de958c8a2fc0958cd76ac79c5df6a6199b7b3345ab6c685ce3 Jul 10 17:43:37.156739 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.156707 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 17:43:37.183849 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.183798 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 17:43:37.184038 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.184008 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.184122 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.184081 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl"] Jul 10 17:43:37.187957 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.186597 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 17:43:37.187957 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.187096 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 17:43:37.187957 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.187333 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 17:43:37.187957 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.187463 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 17:43:37.187957 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.187518 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 17:43:37.187957 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.187630 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 17:43:37.187957 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.187686 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 17:43:37.187957 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.187790 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kn6j5\"" Jul 10 17:43:37.188392 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.188002 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 17:43:37.188392 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.188244 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 17:43:37.188771 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:37.188746 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b70818_5e06_41c7_88fc_a321618a3349.slice/crio-87c1916b6ca57460e8fb0526125f17efb4b55da74dc2521788b18aa93c69476b WatchSource:0}: Error finding container 87c1916b6ca57460e8fb0526125f17efb4b55da74dc2521788b18aa93c69476b: Status 404 returned error can't find the container with id 87c1916b6ca57460e8fb0526125f17efb4b55da74dc2521788b18aa93c69476b Jul 10 17:43:37.228283 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.228212 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66799fcc67-wsrrk" event={"ID":"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74","Type":"ContainerStarted","Data":"55fe88a66d0cf68dd9e9672c39a573e31ef07f64edcdc2abe355979424a1fc59"} Jul 10 17:43:37.232655 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.231765 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" event={"ID":"6a5a8a6f-6afc-4114-998d-4102f3d4e91e","Type":"ContainerStarted","Data":"447b341c582db33032e4df2ba583eb07ea240451c363390b36457dec9bdee28a"} Jul 10 17:43:37.236040 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.236002 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w8nqr" event={"ID":"61455718-aab9-4990-b93b-2c1d7be1532d","Type":"ContainerStarted","Data":"9874a679c82ea1de958c8a2fc0958cd76ac79c5df6a6199b7b3345ab6c685ce3"} Jul 10 17:43:37.239379 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.239324 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" event={"ID":"f1b70818-5e06-41c7-88fc-a321618a3349","Type":"ContainerStarted","Data":"87c1916b6ca57460e8fb0526125f17efb4b55da74dc2521788b18aa93c69476b"} Jul 10 17:43:37.282603 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.282516 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bsrb\" (UniqueName: \"kubernetes.io/projected/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-kube-api-access-4bsrb\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.282944 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.282886 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.283056 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.282997 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.283056 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.283029 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-web-config\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.283056 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.283053 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.283226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.283069 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.283226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.283089 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.283226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.283113 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.283226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.283146 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.283226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.283166 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-config-out\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.283226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.283188 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.283226 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.283212 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.283562 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.283249 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.384549 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.384502 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.384701 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.384553 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-web-config\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.385368 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.384813 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.385368 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.384862 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.385368 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.384899 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.385368 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.384938 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.385368 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.385002 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.385368 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.385033 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-config-out\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.385368 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.385062 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.385368 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.385088 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.385368 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.385140 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.385368 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.385169 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bsrb\" (UniqueName: \"kubernetes.io/projected/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-kube-api-access-4bsrb\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.385368 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.385224 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.385941 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:37.385779 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-trusted-ca-bundle podName:d8359ce3-1500-4d74-92ec-6a2737d1e7c1 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:37.885754502 +0000 UTC m=+267.160081176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1") : configmap references non-existent config key: ca-bundle.crt Jul 10 17:43:37.386659 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.386631 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.386751 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.386690 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.390026 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.389559 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.390026 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.389928 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.390026 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.389935 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-web-config\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.390243 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.390127 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-config-out\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.390352 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.390325 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.390895 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.390853 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.390895 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.390880 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.392068 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.392028 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.392619 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.392599 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.398883 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.398842 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bsrb\" (UniqueName: \"kubernetes.io/projected/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-kube-api-access-4bsrb\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.891388 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.891262 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:37.892332 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:37.892303 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:38.097603 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:38.097332 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:43:38.246817 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:38.246696 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" event={"ID":"f1b70818-5e06-41c7-88fc-a321618a3349","Type":"ContainerStarted","Data":"dfe23cc265de0a6bcca1c876b545280009b40ec7748cff6baa273ff4f230d7bc"} Jul 10 17:43:38.246817 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:38.246767 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" event={"ID":"f1b70818-5e06-41c7-88fc-a321618a3349","Type":"ContainerStarted","Data":"a52d22e114b2297f83ee1070b230ea6c5af3ffd886b47a4f745fc292435f71e8"} Jul 10 17:43:39.963427 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:39.962855 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 17:43:39.967353 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:39.967304 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8359ce3_1500_4d74_92ec_6a2737d1e7c1.slice/crio-915c511b3bad34172776f80e6e46ee9940eddadfea063a061fd7eb6aa9a81787 WatchSource:0}: Error finding container 915c511b3bad34172776f80e6e46ee9940eddadfea063a061fd7eb6aa9a81787: Status 404 returned error can't find the container with id 915c511b3bad34172776f80e6e46ee9940eddadfea063a061fd7eb6aa9a81787 Jul 10 17:43:40.258051 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.258008 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" event={"ID":"6a5a8a6f-6afc-4114-998d-4102f3d4e91e","Type":"ContainerStarted","Data":"11d2ee1847afe88da4e1de5cc6d1728686af86fa9f1ada218398caddc8edd5f7"} Jul 10 17:43:40.258239 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.258089 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" event={"ID":"6a5a8a6f-6afc-4114-998d-4102f3d4e91e","Type":"ContainerStarted","Data":"444ef8a1bd515c59d2ede3831018cd7cc1ede844ea99323a1211fa97539e01fe"} Jul 10 17:43:40.258239 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.258124 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" event={"ID":"6a5a8a6f-6afc-4114-998d-4102f3d4e91e","Type":"ContainerStarted","Data":"51048aab7b08f8a5553bdb376c7ad30687f62b145cf7c2e541458e6491bc6cb0"} Jul 10 17:43:40.259653 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.259624 2556 generic.go:356] "Generic (PLEG): container finished" podID="61455718-aab9-4990-b93b-2c1d7be1532d" containerID="979f9085ffbd6ca093a3c6e5539c023d7f4748465772df155081679dd89b9e83" exitCode=0 Jul 10 17:43:40.259776 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.259709 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w8nqr" event={"ID":"61455718-aab9-4990-b93b-2c1d7be1532d","Type":"ContainerDied","Data":"979f9085ffbd6ca093a3c6e5539c023d7f4748465772df155081679dd89b9e83"} Jul 10 17:43:40.262014 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.261961 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" event={"ID":"f1b70818-5e06-41c7-88fc-a321618a3349","Type":"ContainerStarted","Data":"37b5adfc12160d5ac15cc8725ff907585d2abffd6c10ed0584b1cf76385fc0c5"} Jul 10 17:43:40.263255 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.263236 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerStarted","Data":"915c511b3bad34172776f80e6e46ee9940eddadfea063a061fd7eb6aa9a81787"} Jul 10 17:43:40.266281 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.264909 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66799fcc67-wsrrk" event={"ID":"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74","Type":"ContainerStarted","Data":"6fed9118008e4e5edf11223c0ba070805d220589f19ed53cb23bfc25e6d1187d"} Jul 10 17:43:40.276399 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.276347 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrdnb" podStartSLOduration=1.055511635 podStartE2EDuration="4.276323621s" totalImagesPullingTime="3.220811986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:40.27528878 +0000 UTC m=+269.549615477" watchObservedRunningTime="2026-07-10 17:43:40.276323621 +0000 UTC m=+269.550650317" Jul 10 17:43:40.292884 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.292837 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66799fcc67-wsrrk" podStartSLOduration=1.6930293459999999 podStartE2EDuration="5.292820384s" totalImagesPullingTime="3.599791038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:40.291842175 +0000 UTC m=+269.566168883" watchObservedRunningTime="2026-07-10 17:43:40.292820384 +0000 UTC m=+269.567147078" Jul 10 17:43:40.310645 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.310590 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gt6gl" podStartSLOduration=1.945936139 podStartE2EDuration="4.310573824s" totalImagesPullingTime="2.364637685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:40.309024161 +0000 UTC m=+269.583350859" watchObservedRunningTime="2026-07-10 17:43:40.310573824 +0000 UTC m=+269.584900522" Jul 10 17:43:40.868991 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.868933 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt"] Jul 10 17:43:40.871636 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.871609 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt" Jul 10 17:43:40.874038 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.874009 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 10 17:43:40.874170 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.874046 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tr4ld\"" Jul 10 17:43:40.884558 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:40.884524 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt"] Jul 10 17:43:41.021375 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:41.021341 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gx9w\" (UniqueName: \"kubernetes.io/projected/2aa5cef4-904a-489d-8683-41a01dc7030d-kube-api-access-5gx9w\") pod \"monitoring-plugin-5f68df7f58-9q2mt\" (UID: \"2aa5cef4-904a-489d-8683-41a01dc7030d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt" Jul 10 17:43:41.021723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:41.021400 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2aa5cef4-904a-489d-8683-41a01dc7030d-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-9q2mt\" (UID: \"2aa5cef4-904a-489d-8683-41a01dc7030d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt" Jul 10 17:43:41.122517 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:41.122478 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gx9w\" (UniqueName: \"kubernetes.io/projected/2aa5cef4-904a-489d-8683-41a01dc7030d-kube-api-access-5gx9w\") pod \"monitoring-plugin-5f68df7f58-9q2mt\" (UID: \"2aa5cef4-904a-489d-8683-41a01dc7030d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt" Jul 10 17:43:41.122690 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:41.122531 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2aa5cef4-904a-489d-8683-41a01dc7030d-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-9q2mt\" (UID: \"2aa5cef4-904a-489d-8683-41a01dc7030d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt" Jul 10 17:43:41.125714 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:41.125292 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2aa5cef4-904a-489d-8683-41a01dc7030d-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-9q2mt\" (UID: \"2aa5cef4-904a-489d-8683-41a01dc7030d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt" Jul 10 17:43:41.130465 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:41.130440 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gx9w\" (UniqueName: \"kubernetes.io/projected/2aa5cef4-904a-489d-8683-41a01dc7030d-kube-api-access-5gx9w\") pod \"monitoring-plugin-5f68df7f58-9q2mt\" (UID: \"2aa5cef4-904a-489d-8683-41a01dc7030d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt" Jul 10 17:43:41.184092 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:41.184061 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt" Jul 10 17:43:41.272355 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:41.271987 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w8nqr" event={"ID":"61455718-aab9-4990-b93b-2c1d7be1532d","Type":"ContainerStarted","Data":"f6d9aaf1ebe0ea42f4ad1f45359125754307f9abbc7a8f59a834092af003c2cb"} Jul 10 17:43:41.272355 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:41.272063 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w8nqr" event={"ID":"61455718-aab9-4990-b93b-2c1d7be1532d","Type":"ContainerStarted","Data":"9aebca1a8b5357e027a5e915d9051e0384f932944c0689f696ba1973a0076dfe"} Jul 10 17:43:41.274323 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:41.274294 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerID="7435075d47c991a3bf3befc952fd2ad1adcd1d27a39756ed057f290b9b6c0923" exitCode=0 Jul 10 17:43:41.274457 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:41.274413 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerDied","Data":"7435075d47c991a3bf3befc952fd2ad1adcd1d27a39756ed057f290b9b6c0923"} Jul 10 17:43:41.291200 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:41.291153 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w8nqr" podStartSLOduration=2.436487723 podStartE2EDuration="5.291137363s" totalImagesPullingTime="2.75258466s" totalInitContainerRuntime="102.06498ms" isStatefulPod=true podCreationTimestamp="2026-07-10 17:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:41.290686731 +0000 UTC m=+270.565013428" watchObservedRunningTime="2026-07-10 17:43:41.291137363 +0000 UTC m=+270.565464057" Jul 10 17:43:41.329537 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:41.329426 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt"] Jul 10 17:43:41.332044 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:41.332024 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa5cef4_904a_489d_8683_41a01dc7030d.slice/crio-a5dc485cbf5b7c7ab06426026d9d28daf2e0a9728ecec4e2c75feb621ff96158 WatchSource:0}: Error finding container a5dc485cbf5b7c7ab06426026d9d28daf2e0a9728ecec4e2c75feb621ff96158: Status 404 returned error can't find the container with id a5dc485cbf5b7c7ab06426026d9d28daf2e0a9728ecec4e2c75feb621ff96158 Jul 10 17:43:42.282076 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:42.282024 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt" event={"ID":"2aa5cef4-904a-489d-8683-41a01dc7030d","Type":"ContainerStarted","Data":"a5dc485cbf5b7c7ab06426026d9d28daf2e0a9728ecec4e2c75feb621ff96158"} Jul 10 17:43:43.286790 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:43.286759 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt" event={"ID":"2aa5cef4-904a-489d-8683-41a01dc7030d","Type":"ContainerStarted","Data":"3aef51f0f73756d1d30cc543b07b98f026c3c50929095a70c7a0b334b769ca23"} Jul 10 17:43:43.287159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:43.286908 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt" Jul 10 17:43:43.291859 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:43.291839 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt" Jul 10 17:43:43.302812 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:43.302770 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-9q2mt" podStartSLOduration=1.9646648629999999 podStartE2EDuration="3.302753111s" totalImagesPullingTime="1.338088248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:43.300708037 +0000 UTC m=+272.575034731" watchObservedRunningTime="2026-07-10 17:43:43.302753111 +0000 UTC m=+272.577079824" Jul 10 17:43:44.294178 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:44.294140 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerStarted","Data":"4ad9b9d2bd24741829ff9171de9044c0450f05596849a742db6cb0cb8bd19b63"} Jul 10 17:43:44.294556 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:44.294242 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerStarted","Data":"88244623abc28d6b8531e3a56c2024bf97698430c06cf8c8b8ff996cb471d2dd"} Jul 10 17:43:44.294556 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:44.294269 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerStarted","Data":"9662b0ef6e462e3a0e382f5d66610193cd3ec12b880afa921897b33f2fbe3861"} Jul 10 17:43:44.294556 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:44.294302 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerStarted","Data":"3156b0e6f8d46dffc3ccc1ac53e6c4b154b2f8a2bfa8c9cd2e599b539b2144da"} Jul 10 17:43:44.294556 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:44.294379 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerStarted","Data":"fcaa48d7d7b4b57ed283351eb0a3ea0ded0b46853862726e19f71270a2192be4"} Jul 10 17:43:45.301588 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:45.301549 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerStarted","Data":"a434cf0cc979e9e086589d6821815e08eb5ac01804a2aaeea511f89126c8be85"} Jul 10 17:43:45.330426 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:45.329774 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.449386105 podStartE2EDuration="8.329738587s" totalImagesPullingTime="3.860444527s" totalInitContainerRuntime="19.907955ms" isStatefulPod=false podCreationTimestamp="2026-07-10 17:43:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:45.327839809 +0000 UTC m=+274.602166503" watchObservedRunningTime="2026-07-10 17:43:45.329738587 +0000 UTC m=+274.604065282" Jul 10 17:43:46.037176 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:46.037133 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:46.037351 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:46.037225 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:46.042309 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:46.042283 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:46.309391 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:46.309306 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:43:52.877868 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:43:52.877809 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8dblb" podUID="1f6a4ab1-f10d-4111-812a-90d3f35f3484" Jul 10 17:43:53.332413 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:53.332331 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8dblb" Jul 10 17:43:53.831270 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:53.831232 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66799fcc67-wsrrk"] Jul 10 17:43:55.957453 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:55.957409 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:43:55.959928 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:55.959906 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f6a4ab1-f10d-4111-812a-90d3f35f3484-metrics-tls\") pod \"dns-default-8dblb\" (UID: \"1f6a4ab1-f10d-4111-812a-90d3f35f3484\") " pod="openshift-dns/dns-default-8dblb" Jul 10 17:43:56.035490 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:56.035456 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dbkj9\"" Jul 10 17:43:56.043471 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:56.043450 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8dblb" Jul 10 17:43:56.216601 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:56.216579 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8dblb"] Jul 10 17:43:56.218779 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:43:56.218751 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6a4ab1_f10d_4111_812a_90d3f35f3484.slice/crio-c300b99241a2873c5f297844076e649db16fa1865c63b3e96bab90cc9000dd6e WatchSource:0}: Error finding container c300b99241a2873c5f297844076e649db16fa1865c63b3e96bab90cc9000dd6e: Status 404 returned error can't find the container with id c300b99241a2873c5f297844076e649db16fa1865c63b3e96bab90cc9000dd6e Jul 10 17:43:56.344602 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:56.344567 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8dblb" event={"ID":"1f6a4ab1-f10d-4111-812a-90d3f35f3484","Type":"ContainerStarted","Data":"c300b99241a2873c5f297844076e649db16fa1865c63b3e96bab90cc9000dd6e"} Jul 10 17:43:58.354776 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:58.354737 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8dblb" event={"ID":"1f6a4ab1-f10d-4111-812a-90d3f35f3484","Type":"ContainerStarted","Data":"9500c0bc7b929566dfd54f3a0f40a6472c7f415b9ffa8d25e8f8107999966c0b"} Jul 10 17:43:58.355213 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:58.354817 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8dblb" event={"ID":"1f6a4ab1-f10d-4111-812a-90d3f35f3484","Type":"ContainerStarted","Data":"0bbd71492fa1d51b561248ed5be8366db7e1d1c03164de2c5c8394bdb200e16b"} Jul 10 17:43:58.355213 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:58.354862 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8dblb" Jul 10 17:43:58.373046 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:43:58.372986 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8dblb" podStartSLOduration=250.787721637 podStartE2EDuration="4m12.372952167s" totalImagesPullingTime="1.58523053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:58.370667394 +0000 UTC m=+287.644994088" watchObservedRunningTime="2026-07-10 17:43:58.372952167 +0000 UTC m=+287.647278904" Jul 10 17:44:08.362769 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:08.362734 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8dblb" Jul 10 17:44:11.084995 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:11.084948 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 17:44:11.085464 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:11.085291 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 17:44:11.095585 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:11.095564 2556 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 17:44:18.854190 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:18.854121 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66799fcc67-wsrrk" podUID="9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74" containerName="console" containerID="cri-o://6fed9118008e4e5edf11223c0ba070805d220589f19ed53cb23bfc25e6d1187d" gracePeriod=15 Jul 10 17:44:19.098248 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.098223 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:44:19.258049 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.257954 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-config\") pod \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " Jul 10 17:44:19.258049 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.258011 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfnlz\" (UniqueName: \"kubernetes.io/projected/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-kube-api-access-sfnlz\") pod \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " Jul 10 17:44:19.258267 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.258078 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-oauth-config\") pod \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " Jul 10 17:44:19.258267 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.258099 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-trusted-ca-bundle\") pod \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " Jul 10 17:44:19.258267 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.258118 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-serving-cert\") pod \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " Jul 10 17:44:19.258267 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.258132 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-oauth-serving-cert\") pod \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " Jul 10 17:44:19.258267 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.258153 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-service-ca\") pod \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\" (UID: \"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74\") " Jul 10 17:44:19.258513 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.258277 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-config" (OuterVolumeSpecName: "console-config") pod "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74" (UID: "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:44:19.258594 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.258562 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74" (UID: "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:44:19.258674 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.258650 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74" (UID: "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:44:19.258674 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.258662 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-service-ca" (OuterVolumeSpecName: "service-ca") pod "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74" (UID: "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:44:19.260506 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.260479 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74" (UID: "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:44:19.260506 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.260490 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74" (UID: "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:44:19.260639 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.260607 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-kube-api-access-sfnlz" (OuterVolumeSpecName: "kube-api-access-sfnlz") pod "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74" (UID: "9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74"). InnerVolumeSpecName "kube-api-access-sfnlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:44:19.359370 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.359338 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-oauth-config\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:19.359370 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.359365 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-trusted-ca-bundle\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:19.359370 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.359375 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-serving-cert\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:19.359625 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.359384 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-oauth-serving-cert\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:19.359625 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.359394 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-service-ca\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:19.359625 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.359403 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-console-config\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:19.359625 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.359412 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfnlz\" (UniqueName: \"kubernetes.io/projected/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74-kube-api-access-sfnlz\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:19.443951 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.443915 2556 generic.go:356] "Generic (PLEG): container finished" podID="9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74" containerID="6fed9118008e4e5edf11223c0ba070805d220589f19ed53cb23bfc25e6d1187d" exitCode=0 Jul 10 17:44:19.444139 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.443962 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66799fcc67-wsrrk" Jul 10 17:44:19.444139 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.444006 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66799fcc67-wsrrk" event={"ID":"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74","Type":"ContainerDied","Data":"6fed9118008e4e5edf11223c0ba070805d220589f19ed53cb23bfc25e6d1187d"} Jul 10 17:44:19.444139 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.444058 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66799fcc67-wsrrk" event={"ID":"9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74","Type":"ContainerDied","Data":"55fe88a66d0cf68dd9e9672c39a573e31ef07f64edcdc2abe355979424a1fc59"} Jul 10 17:44:19.444139 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.444082 2556 scope.go:117] "RemoveContainer" containerID="6fed9118008e4e5edf11223c0ba070805d220589f19ed53cb23bfc25e6d1187d" Jul 10 17:44:19.452829 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.452811 2556 scope.go:117] "RemoveContainer" containerID="6fed9118008e4e5edf11223c0ba070805d220589f19ed53cb23bfc25e6d1187d" Jul 10 17:44:19.453144 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:44:19.453122 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fed9118008e4e5edf11223c0ba070805d220589f19ed53cb23bfc25e6d1187d\": container with ID starting with 6fed9118008e4e5edf11223c0ba070805d220589f19ed53cb23bfc25e6d1187d not found: ID does not exist" containerID="6fed9118008e4e5edf11223c0ba070805d220589f19ed53cb23bfc25e6d1187d" Jul 10 17:44:19.453252 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.453151 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fed9118008e4e5edf11223c0ba070805d220589f19ed53cb23bfc25e6d1187d"} err="failed to get container status \"6fed9118008e4e5edf11223c0ba070805d220589f19ed53cb23bfc25e6d1187d\": rpc error: code = NotFound desc = could not find container \"6fed9118008e4e5edf11223c0ba070805d220589f19ed53cb23bfc25e6d1187d\": container with ID starting with 6fed9118008e4e5edf11223c0ba070805d220589f19ed53cb23bfc25e6d1187d not found: ID does not exist" Jul 10 17:44:19.465251 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.465173 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66799fcc67-wsrrk"] Jul 10 17:44:19.467724 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:19.467695 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66799fcc67-wsrrk"] Jul 10 17:44:21.253990 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:21.253936 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74" path="/var/lib/kubelet/pods/9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74/volumes" Jul 10 17:44:33.503802 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:33.503711 2556 generic.go:356] "Generic (PLEG): container finished" podID="71a927f1-68cc-4c42-9b68-eed6282b657b" containerID="7afa507232d8500ca9742658e4051959dfa6a628fbe5fb7cc1054de261b6b19a" exitCode=0 Jul 10 17:44:33.503802 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:33.503785 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" event={"ID":"71a927f1-68cc-4c42-9b68-eed6282b657b","Type":"ContainerDied","Data":"7afa507232d8500ca9742658e4051959dfa6a628fbe5fb7cc1054de261b6b19a"} Jul 10 17:44:33.504367 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:33.504212 2556 scope.go:117] "RemoveContainer" containerID="7afa507232d8500ca9742658e4051959dfa6a628fbe5fb7cc1054de261b6b19a" Jul 10 17:44:34.509516 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:34.509481 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x5kg4" event={"ID":"71a927f1-68cc-4c42-9b68-eed6282b657b","Type":"ContainerStarted","Data":"1b2c9d182af2d296a3d5decd8847f7b0f1ad8575d1486126577376b1a8e33bc9"} Jul 10 17:44:56.467196 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.467159 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 17:44:56.467708 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.467447 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="alertmanager" containerID="cri-o://fcaa48d7d7b4b57ed283351eb0a3ea0ded0b46853862726e19f71270a2192be4" gracePeriod=120 Jul 10 17:44:56.467708 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.467538 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="kube-rbac-proxy-metric" containerID="cri-o://4ad9b9d2bd24741829ff9171de9044c0450f05596849a742db6cb0cb8bd19b63" gracePeriod=120 Jul 10 17:44:56.467708 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.467581 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="kube-rbac-proxy-web" containerID="cri-o://9662b0ef6e462e3a0e382f5d66610193cd3ec12b880afa921897b33f2fbe3861" gracePeriod=120 Jul 10 17:44:56.467708 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.467629 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="kube-rbac-proxy" containerID="cri-o://88244623abc28d6b8531e3a56c2024bf97698430c06cf8c8b8ff996cb471d2dd" gracePeriod=120 Jul 10 17:44:56.467708 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.467626 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="prom-label-proxy" containerID="cri-o://a434cf0cc979e9e086589d6821815e08eb5ac01804a2aaeea511f89126c8be85" gracePeriod=120 Jul 10 17:44:56.467708 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.467582 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="config-reloader" containerID="cri-o://3156b0e6f8d46dffc3ccc1ac53e6c4b154b2f8a2bfa8c9cd2e599b539b2144da" gracePeriod=120 Jul 10 17:44:56.596136 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.596109 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerID="a434cf0cc979e9e086589d6821815e08eb5ac01804a2aaeea511f89126c8be85" exitCode=0 Jul 10 17:44:56.596136 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.596132 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerID="4ad9b9d2bd24741829ff9171de9044c0450f05596849a742db6cb0cb8bd19b63" exitCode=0 Jul 10 17:44:56.596136 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.596139 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerID="88244623abc28d6b8531e3a56c2024bf97698430c06cf8c8b8ff996cb471d2dd" exitCode=0 Jul 10 17:44:56.596305 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.596144 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerID="3156b0e6f8d46dffc3ccc1ac53e6c4b154b2f8a2bfa8c9cd2e599b539b2144da" exitCode=0 Jul 10 17:44:56.596305 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.596149 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerID="fcaa48d7d7b4b57ed283351eb0a3ea0ded0b46853862726e19f71270a2192be4" exitCode=0 Jul 10 17:44:56.596305 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.596177 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerDied","Data":"a434cf0cc979e9e086589d6821815e08eb5ac01804a2aaeea511f89126c8be85"} Jul 10 17:44:56.596305 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.596239 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerDied","Data":"4ad9b9d2bd24741829ff9171de9044c0450f05596849a742db6cb0cb8bd19b63"} Jul 10 17:44:56.596305 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.596266 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerDied","Data":"88244623abc28d6b8531e3a56c2024bf97698430c06cf8c8b8ff996cb471d2dd"} Jul 10 17:44:56.596305 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.596290 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerDied","Data":"3156b0e6f8d46dffc3ccc1ac53e6c4b154b2f8a2bfa8c9cd2e599b539b2144da"} Jul 10 17:44:56.596478 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:56.596312 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerDied","Data":"fcaa48d7d7b4b57ed283351eb0a3ea0ded0b46853862726e19f71270a2192be4"} Jul 10 17:44:57.603931 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.603901 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerID="9662b0ef6e462e3a0e382f5d66610193cd3ec12b880afa921897b33f2fbe3861" exitCode=0 Jul 10 17:44:57.604322 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.603994 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerDied","Data":"9662b0ef6e462e3a0e382f5d66610193cd3ec12b880afa921897b33f2fbe3861"} Jul 10 17:44:57.720632 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.720604 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:57.772322 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772269 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " Jul 10 17:44:57.772522 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772380 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-trusted-ca-bundle\") pod \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " Jul 10 17:44:57.772582 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772514 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bsrb\" (UniqueName: \"kubernetes.io/projected/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-kube-api-access-4bsrb\") pod \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " Jul 10 17:44:57.772636 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772602 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-main-tls\") pod \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " Jul 10 17:44:57.772691 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772654 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-config-volume\") pod \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " Jul 10 17:44:57.772744 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772698 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-tls-assets\") pod \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " Jul 10 17:44:57.772794 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772746 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " Jul 10 17:44:57.772794 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772776 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-config-out\") pod \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " Jul 10 17:44:57.772893 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772807 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-web-config\") pod \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " Jul 10 17:44:57.772893 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772833 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-main-db\") pod \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " Jul 10 17:44:57.772893 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772876 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d8359ce3-1500-4d74-92ec-6a2737d1e7c1" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:44:57.773097 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772885 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-cluster-tls-config\") pod \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " Jul 10 17:44:57.773097 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772943 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy\") pod \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " Jul 10 17:44:57.773097 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.772989 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-metrics-client-ca\") pod \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\" (UID: \"d8359ce3-1500-4d74-92ec-6a2737d1e7c1\") " Jul 10 17:44:57.773277 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.773256 2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:57.773890 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.773863 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d8359ce3-1500-4d74-92ec-6a2737d1e7c1" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:44:57.776065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.775567 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-kube-api-access-4bsrb" (OuterVolumeSpecName: "kube-api-access-4bsrb") pod "d8359ce3-1500-4d74-92ec-6a2737d1e7c1" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1"). InnerVolumeSpecName "kube-api-access-4bsrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:44:57.776065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.775607 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d8359ce3-1500-4d74-92ec-6a2737d1e7c1" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:44:57.776065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.775770 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d8359ce3-1500-4d74-92ec-6a2737d1e7c1" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:44:57.776462 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.776327 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-config-volume" (OuterVolumeSpecName: "config-volume") pod "d8359ce3-1500-4d74-92ec-6a2737d1e7c1" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:44:57.776462 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.776343 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d8359ce3-1500-4d74-92ec-6a2737d1e7c1" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:44:57.776613 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.776592 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d8359ce3-1500-4d74-92ec-6a2737d1e7c1" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:44:57.776687 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.776637 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d8359ce3-1500-4d74-92ec-6a2737d1e7c1" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:44:57.777531 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.777504 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-config-out" (OuterVolumeSpecName: "config-out") pod "d8359ce3-1500-4d74-92ec-6a2737d1e7c1" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:44:57.777831 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.777811 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d8359ce3-1500-4d74-92ec-6a2737d1e7c1" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:44:57.782295 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.782267 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d8359ce3-1500-4d74-92ec-6a2737d1e7c1" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:44:57.790920 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.790888 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-web-config" (OuterVolumeSpecName: "web-config") pod "d8359ce3-1500-4d74-92ec-6a2737d1e7c1" (UID: "d8359ce3-1500-4d74-92ec-6a2737d1e7c1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:44:57.873938 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.873908 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:57.873938 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.873937 2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-config-out\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:57.874159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.873947 2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-web-config\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:57.874159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.873956 2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-alertmanager-main-db\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:57.874159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.873986 2556 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-cluster-tls-config\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:57.874159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.873995 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:57.874159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.874004 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-metrics-client-ca\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:57.874159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.874013 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:57.874159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.874022 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bsrb\" (UniqueName: \"kubernetes.io/projected/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-kube-api-access-4bsrb\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:57.874159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.874031 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-secret-alertmanager-main-tls\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:57.874159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.874039 2556 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-config-volume\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:57.874159 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:57.874046 2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8359ce3-1500-4d74-92ec-6a2737d1e7c1-tls-assets\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:44:58.611002 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.610954 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8359ce3-1500-4d74-92ec-6a2737d1e7c1","Type":"ContainerDied","Data":"915c511b3bad34172776f80e6e46ee9940eddadfea063a061fd7eb6aa9a81787"} Jul 10 17:44:58.611475 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.611033 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.611475 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.611048 2556 scope.go:117] "RemoveContainer" containerID="a434cf0cc979e9e086589d6821815e08eb5ac01804a2aaeea511f89126c8be85" Jul 10 17:44:58.619065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.618945 2556 scope.go:117] "RemoveContainer" containerID="4ad9b9d2bd24741829ff9171de9044c0450f05596849a742db6cb0cb8bd19b63" Jul 10 17:44:58.626528 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.626510 2556 scope.go:117] "RemoveContainer" containerID="88244623abc28d6b8531e3a56c2024bf97698430c06cf8c8b8ff996cb471d2dd" Jul 10 17:44:58.633442 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.633419 2556 scope.go:117] "RemoveContainer" containerID="9662b0ef6e462e3a0e382f5d66610193cd3ec12b880afa921897b33f2fbe3861" Jul 10 17:44:58.635758 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.635735 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 17:44:58.640506 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.640481 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 17:44:58.642133 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.642109 2556 scope.go:117] "RemoveContainer" containerID="3156b0e6f8d46dffc3ccc1ac53e6c4b154b2f8a2bfa8c9cd2e599b539b2144da" Jul 10 17:44:58.649528 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.649513 2556 scope.go:117] "RemoveContainer" containerID="fcaa48d7d7b4b57ed283351eb0a3ea0ded0b46853862726e19f71270a2192be4" Jul 10 17:44:58.657260 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.657234 2556 scope.go:117] "RemoveContainer" containerID="7435075d47c991a3bf3befc952fd2ad1adcd1d27a39756ed057f290b9b6c0923" Jul 10 17:44:58.667257 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667235 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 17:44:58.667536 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667524 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74" containerName="console" Jul 10 17:44:58.667588 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667539 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74" containerName="console" Jul 10 17:44:58.667588 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667550 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="kube-rbac-proxy-metric" Jul 10 17:44:58.667588 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667555 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="kube-rbac-proxy-metric" Jul 10 17:44:58.667588 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667563 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="prom-label-proxy" Jul 10 17:44:58.667588 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667568 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="prom-label-proxy" Jul 10 17:44:58.667588 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667574 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="alertmanager" Jul 10 17:44:58.667588 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667580 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="alertmanager" Jul 10 17:44:58.667588 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667588 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="kube-rbac-proxy" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667593 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="kube-rbac-proxy" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667600 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="init-config-reloader" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667606 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="init-config-reloader" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667612 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="kube-rbac-proxy-web" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667617 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="kube-rbac-proxy-web" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667627 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="config-reloader" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667632 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="config-reloader" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667676 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="kube-rbac-proxy-web" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667687 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc0ca44-1c79-4fb6-abb5-15c92c8e9b74" containerName="console" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667692 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="kube-rbac-proxy" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667697 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="kube-rbac-proxy-metric" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667702 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="prom-label-proxy" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667708 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="alertmanager" Jul 10 17:44:58.667816 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.667713 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" containerName="config-reloader" Jul 10 17:44:58.672641 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.672623 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.675196 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.675171 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 17:44:58.675300 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.675239 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 17:44:58.675300 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.675248 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 17:44:58.675300 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.675264 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 17:44:58.675300 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.675264 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kn6j5\"" Jul 10 17:44:58.675567 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.675550 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 17:44:58.675629 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.675575 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 17:44:58.675629 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.675586 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 17:44:58.675629 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.675615 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 17:44:58.681262 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.681241 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 17:44:58.684879 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.684857 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 17:44:58.781803 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.781770 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.781992 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.781825 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-config-out\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.781992 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.781862 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.781992 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.781881 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.781992 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.781902 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.781992 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.781917 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.781992 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.781939 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.782230 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.782005 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.782230 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.782042 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.782230 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.782062 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-web-config\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.782230 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.782080 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.782230 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.782127 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glf2b\" (UniqueName: \"kubernetes.io/projected/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-kube-api-access-glf2b\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.782230 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.782166 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.882762 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.882728 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.882762 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.882769 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.883065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.882813 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-config-out\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.883065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.882829 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.883065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.882847 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.883065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.882867 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.883065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.882890 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.883065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.882913 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.883065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.882944 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.883065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.883000 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.883065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.883023 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-web-config\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.883065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.883047 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.883065 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.883068 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glf2b\" (UniqueName: \"kubernetes.io/projected/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-kube-api-access-glf2b\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.884011 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.883672 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.884122 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.884061 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.885306 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.885281 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.886064 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.885906 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-config-out\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.886186 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.886162 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.886559 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.886530 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.886658 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.886537 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.886658 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.886541 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.886658 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.886588 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.887237 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.887212 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.887871 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.887854 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-web-config\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.888583 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.888565 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.890654 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.890633 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glf2b\" (UniqueName: \"kubernetes.io/projected/c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905-kube-api-access-glf2b\") pod \"alertmanager-main-0\" (UID: \"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:58.983558 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:58.983517 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 17:44:59.130803 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:59.130769 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 17:44:59.134666 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:44:59.134605 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f12f0d_1cf2_4cd5_8b9b_392a8bf28905.slice/crio-f20bcff3e018b7dfec432a99db008b547027d5b64edcdd395de87c3356265090 WatchSource:0}: Error finding container f20bcff3e018b7dfec432a99db008b547027d5b64edcdd395de87c3356265090: Status 404 returned error can't find the container with id f20bcff3e018b7dfec432a99db008b547027d5b64edcdd395de87c3356265090 Jul 10 17:44:59.254282 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:59.254248 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8359ce3-1500-4d74-92ec-6a2737d1e7c1" path="/var/lib/kubelet/pods/d8359ce3-1500-4d74-92ec-6a2737d1e7c1/volumes" Jul 10 17:44:59.617790 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:59.617757 2556 generic.go:356] "Generic (PLEG): container finished" podID="c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905" containerID="28eac6192e141d37ca3ecff56621d9976c0346cc1f72aad8f2576aa77d002edd" exitCode=0 Jul 10 17:44:59.618305 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:59.617805 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905","Type":"ContainerDied","Data":"28eac6192e141d37ca3ecff56621d9976c0346cc1f72aad8f2576aa77d002edd"} Jul 10 17:44:59.618305 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:44:59.617854 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905","Type":"ContainerStarted","Data":"f20bcff3e018b7dfec432a99db008b547027d5b64edcdd395de87c3356265090"} Jul 10 17:45:00.625297 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:00.625257 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905","Type":"ContainerStarted","Data":"9d50eb0d13d18b31b8f981e19a9f5ae35317f1b3b02816c80abc126a9d9f263f"} Jul 10 17:45:00.625679 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:00.625372 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905","Type":"ContainerStarted","Data":"c37e59c5749d15df6550d19a48c05bfe8f8710f8a76afb2bf9b78e2148fdb8d9"} Jul 10 17:45:00.625679 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:00.625415 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905","Type":"ContainerStarted","Data":"2e48f456126cf08a01358c2ecd4e0c024fb9f11567f1c841f32b6380e5d8b8fe"} Jul 10 17:45:00.625679 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:00.625448 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905","Type":"ContainerStarted","Data":"063b23bbc9553ee89ec8090289b351425434e24fb85df13d417cd5e8fd12bbd8"} Jul 10 17:45:00.625679 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:00.625483 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905","Type":"ContainerStarted","Data":"db2a71e7b5e57ecfaad0d8aafaa18cc006b0406f992c23a93f45790376a93875"} Jul 10 17:45:00.625679 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:00.625523 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905","Type":"ContainerStarted","Data":"94fdfcadb8b52e7c95fca1c5b6de3ea151b5e943837958f066207838dc6f900e"} Jul 10 17:45:00.653326 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:00.653277 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.636917216 podStartE2EDuration="2.653260175s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.342959ms" isStatefulPod=false podCreationTimestamp="2026-07-10 17:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:45:00.650798798 +0000 UTC m=+349.925125505" watchObservedRunningTime="2026-07-10 17:45:00.653260175 +0000 UTC m=+349.927586871" Jul 10 17:45:19.773212 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.773174 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9fb8465f8-sc2fd"] Jul 10 17:45:19.778132 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.778110 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.780784 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.780758 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 17:45:19.780928 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.780810 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 17:45:19.780928 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.780903 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 17:45:19.781076 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.780928 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 17:45:19.782078 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.782057 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z6jhd\"" Jul 10 17:45:19.782078 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.782079 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 17:45:19.782265 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.782132 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 17:45:19.782265 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.782185 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 17:45:19.787202 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.787176 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9fb8465f8-sc2fd"] Jul 10 17:45:19.787326 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.787294 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 17:45:19.854777 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.854744 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-service-ca\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.854945 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.854791 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj75r\" (UniqueName: \"kubernetes.io/projected/b5046f8d-71be-4389-b77f-0c4482cbc6b1-kube-api-access-tj75r\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.854945 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.854816 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-config\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.854945 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.854880 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-serving-cert\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.854945 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.854930 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-trusted-ca-bundle\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.855122 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.854951 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-oauth-config\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.855122 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.855021 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-oauth-serving-cert\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.955600 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.955563 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-service-ca\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.955754 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.955626 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj75r\" (UniqueName: \"kubernetes.io/projected/b5046f8d-71be-4389-b77f-0c4482cbc6b1-kube-api-access-tj75r\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.955754 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.955731 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-config\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.955850 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.955791 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-serving-cert\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.955901 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.955863 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-trusted-ca-bundle\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.955901 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.955889 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-oauth-config\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.956044 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.955922 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-oauth-serving-cert\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.956376 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.956354 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-service-ca\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.956664 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.956469 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-config\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.956664 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.956544 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-oauth-serving-cert\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.956781 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.956678 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-trusted-ca-bundle\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.958343 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.958325 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-serving-cert\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.958448 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.958379 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-oauth-config\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:19.963421 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:19.963392 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj75r\" (UniqueName: \"kubernetes.io/projected/b5046f8d-71be-4389-b77f-0c4482cbc6b1-kube-api-access-tj75r\") pod \"console-9fb8465f8-sc2fd\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:20.090256 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:20.090150 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:20.222648 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:20.222625 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9fb8465f8-sc2fd"] Jul 10 17:45:20.225093 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:45:20.225068 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5046f8d_71be_4389_b77f_0c4482cbc6b1.slice/crio-d45671fc038b9d08c3545287ac5193f0ec86c5a1eff11f3a6de46af2c5bb4040 WatchSource:0}: Error finding container d45671fc038b9d08c3545287ac5193f0ec86c5a1eff11f3a6de46af2c5bb4040: Status 404 returned error can't find the container with id d45671fc038b9d08c3545287ac5193f0ec86c5a1eff11f3a6de46af2c5bb4040 Jul 10 17:45:20.711148 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:20.711114 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fb8465f8-sc2fd" event={"ID":"b5046f8d-71be-4389-b77f-0c4482cbc6b1","Type":"ContainerStarted","Data":"69220a0f54c0f1b32629d547781c5ffebdfec03e3335e7506b259021b7919d05"} Jul 10 17:45:20.711316 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:20.711162 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fb8465f8-sc2fd" event={"ID":"b5046f8d-71be-4389-b77f-0c4482cbc6b1","Type":"ContainerStarted","Data":"d45671fc038b9d08c3545287ac5193f0ec86c5a1eff11f3a6de46af2c5bb4040"} Jul 10 17:45:20.730105 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:20.730052 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9fb8465f8-sc2fd" podStartSLOduration=1.730035366 podStartE2EDuration="1.730035366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:45:20.727789552 +0000 UTC m=+370.002116252" watchObservedRunningTime="2026-07-10 17:45:20.730035366 +0000 UTC m=+370.004362060" Jul 10 17:45:30.090407 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:30.090348 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:30.090886 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:30.090436 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:30.095267 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:30.095244 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:45:30.752928 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:45:30.752898 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:46:52.009952 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.009870 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-l9spz"] Jul 10 17:46:52.012235 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.012219 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-l9spz" Jul 10 17:46:52.015054 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.015018 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 17:46:52.015209 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.015023 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 17:46:52.016081 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.016063 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-f898w\"" Jul 10 17:46:52.021724 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.021704 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-l9spz"] Jul 10 17:46:52.107707 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.107670 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z88bf\" (UniqueName: \"kubernetes.io/projected/20121a75-a708-4465-8132-d9f61f56b824-kube-api-access-z88bf\") pod \"cert-manager-cainjector-559cdd8c87-l9spz\" (UID: \"20121a75-a708-4465-8132-d9f61f56b824\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-l9spz" Jul 10 17:46:52.107707 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.107710 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20121a75-a708-4465-8132-d9f61f56b824-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-l9spz\" (UID: \"20121a75-a708-4465-8132-d9f61f56b824\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-l9spz" Jul 10 17:46:52.208578 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.208546 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z88bf\" (UniqueName: \"kubernetes.io/projected/20121a75-a708-4465-8132-d9f61f56b824-kube-api-access-z88bf\") pod \"cert-manager-cainjector-559cdd8c87-l9spz\" (UID: \"20121a75-a708-4465-8132-d9f61f56b824\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-l9spz" Jul 10 17:46:52.208578 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.208581 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20121a75-a708-4465-8132-d9f61f56b824-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-l9spz\" (UID: \"20121a75-a708-4465-8132-d9f61f56b824\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-l9spz" Jul 10 17:46:52.217132 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.217101 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20121a75-a708-4465-8132-d9f61f56b824-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-l9spz\" (UID: \"20121a75-a708-4465-8132-d9f61f56b824\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-l9spz" Jul 10 17:46:52.217252 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.217160 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z88bf\" (UniqueName: \"kubernetes.io/projected/20121a75-a708-4465-8132-d9f61f56b824-kube-api-access-z88bf\") pod \"cert-manager-cainjector-559cdd8c87-l9spz\" (UID: \"20121a75-a708-4465-8132-d9f61f56b824\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-l9spz" Jul 10 17:46:52.321606 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.321511 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-l9spz" Jul 10 17:46:52.453795 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.453770 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-l9spz"] Jul 10 17:46:52.456370 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:46:52.456343 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20121a75_a708_4465_8132_d9f61f56b824.slice/crio-6321b19fb5fdf81bd916d0a71d7d6c4166152ec605187c8f97c1f46e2fc7c3a4 WatchSource:0}: Error finding container 6321b19fb5fdf81bd916d0a71d7d6c4166152ec605187c8f97c1f46e2fc7c3a4: Status 404 returned error can't find the container with id 6321b19fb5fdf81bd916d0a71d7d6c4166152ec605187c8f97c1f46e2fc7c3a4 Jul 10 17:46:52.458355 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:52.458338 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:46:53.077668 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:53.077634 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-l9spz" event={"ID":"20121a75-a708-4465-8132-d9f61f56b824","Type":"ContainerStarted","Data":"6321b19fb5fdf81bd916d0a71d7d6c4166152ec605187c8f97c1f46e2fc7c3a4"} Jul 10 17:46:56.093772 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:56.093674 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-l9spz" event={"ID":"20121a75-a708-4465-8132-d9f61f56b824","Type":"ContainerStarted","Data":"3cfe58c52c8473917b1db656c23c423140be358ba89a28f9ce006a91b936dfa8"} Jul 10 17:46:56.113999 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:46:56.111388 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-l9spz" podStartSLOduration=1.8201959909999998 podStartE2EDuration="5.111363793s" totalImagesPullingTime="3.291167802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:46:56.109046447 +0000 UTC m=+465.383373139" watchObservedRunningTime="2026-07-10 17:46:56.111363793 +0000 UTC m=+465.385690489" Jul 10 17:47:01.592887 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:01.592851 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj"] Jul 10 17:47:01.596232 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:01.596215 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj" Jul 10 17:47:01.598856 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:01.598831 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:47:01.598988 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:01.598831 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-qsmsw\"" Jul 10 17:47:01.599792 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:01.599776 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 17:47:01.607262 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:01.607234 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj"] Jul 10 17:47:01.692284 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:01.692234 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2rn7\" (UniqueName: \"kubernetes.io/projected/f131c16c-eaee-4ace-8c55-a2d8e976fb9c-kube-api-access-s2rn7\") pod \"openshift-lws-operator-fd8ccff4c-qklzj\" (UID: \"f131c16c-eaee-4ace-8c55-a2d8e976fb9c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj" Jul 10 17:47:01.692480 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:01.692372 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f131c16c-eaee-4ace-8c55-a2d8e976fb9c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qklzj\" (UID: \"f131c16c-eaee-4ace-8c55-a2d8e976fb9c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj" Jul 10 17:47:01.793795 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:01.793748 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2rn7\" (UniqueName: \"kubernetes.io/projected/f131c16c-eaee-4ace-8c55-a2d8e976fb9c-kube-api-access-s2rn7\") pod \"openshift-lws-operator-fd8ccff4c-qklzj\" (UID: \"f131c16c-eaee-4ace-8c55-a2d8e976fb9c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj" Jul 10 17:47:01.793995 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:01.793827 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f131c16c-eaee-4ace-8c55-a2d8e976fb9c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qklzj\" (UID: \"f131c16c-eaee-4ace-8c55-a2d8e976fb9c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj" Jul 10 17:47:01.794205 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:01.794185 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f131c16c-eaee-4ace-8c55-a2d8e976fb9c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qklzj\" (UID: \"f131c16c-eaee-4ace-8c55-a2d8e976fb9c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj" Jul 10 17:47:01.802350 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:01.802307 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2rn7\" (UniqueName: \"kubernetes.io/projected/f131c16c-eaee-4ace-8c55-a2d8e976fb9c-kube-api-access-s2rn7\") pod \"openshift-lws-operator-fd8ccff4c-qklzj\" (UID: \"f131c16c-eaee-4ace-8c55-a2d8e976fb9c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj" Jul 10 17:47:01.907281 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:01.907236 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj" Jul 10 17:47:02.042689 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:02.042663 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj"] Jul 10 17:47:02.045494 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:47:02.045470 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf131c16c_eaee_4ace_8c55_a2d8e976fb9c.slice/crio-43dc326baf89db8e81841ecc9f18349a2fface5fa2fef09ad2716a6bf31496ad WatchSource:0}: Error finding container 43dc326baf89db8e81841ecc9f18349a2fface5fa2fef09ad2716a6bf31496ad: Status 404 returned error can't find the container with id 43dc326baf89db8e81841ecc9f18349a2fface5fa2fef09ad2716a6bf31496ad Jul 10 17:47:02.120015 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:02.119962 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj" event={"ID":"f131c16c-eaee-4ace-8c55-a2d8e976fb9c","Type":"ContainerStarted","Data":"43dc326baf89db8e81841ecc9f18349a2fface5fa2fef09ad2716a6bf31496ad"} Jul 10 17:47:26.222716 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:26.222639 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj" event={"ID":"f131c16c-eaee-4ace-8c55-a2d8e976fb9c","Type":"ContainerStarted","Data":"517a0e61c5b4662074c558c0f2a77424cc6bffba858d5d79456c53d707d36b27"} Jul 10 17:47:26.238740 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:26.238687 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qklzj" podStartSLOduration=1.359001211 podStartE2EDuration="25.238671528s" totalImagesPullingTime="23.879670317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:47:26.237334858 +0000 UTC m=+495.511661554" watchObservedRunningTime="2026-07-10 17:47:26.238671528 +0000 UTC m=+495.512998224" Jul 10 17:47:33.231808 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.231775 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx"] Jul 10 17:47:33.234912 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.234895 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.238641 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.238623 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 17:47:33.238718 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.238623 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 17:47:33.238911 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.238893 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cgnpf\"" Jul 10 17:47:33.238999 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.238935 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 17:47:33.247271 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.247248 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx"] Jul 10 17:47:33.258723 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.258696 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4q4t\" (UniqueName: \"kubernetes.io/projected/6199be07-663c-4792-afaa-ca98a12c9902-kube-api-access-r4q4t\") pod \"lws-controller-manager-7659478ddc-zzqfx\" (UID: \"6199be07-663c-4792-afaa-ca98a12c9902\") " pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.258842 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.258735 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6199be07-663c-4792-afaa-ca98a12c9902-cert\") pod \"lws-controller-manager-7659478ddc-zzqfx\" (UID: \"6199be07-663c-4792-afaa-ca98a12c9902\") " pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.258842 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.258755 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6199be07-663c-4792-afaa-ca98a12c9902-manager-config\") pod \"lws-controller-manager-7659478ddc-zzqfx\" (UID: \"6199be07-663c-4792-afaa-ca98a12c9902\") " pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.258842 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.258785 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6199be07-663c-4792-afaa-ca98a12c9902-metrics-cert\") pod \"lws-controller-manager-7659478ddc-zzqfx\" (UID: \"6199be07-663c-4792-afaa-ca98a12c9902\") " pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.359481 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.359442 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q4t\" (UniqueName: \"kubernetes.io/projected/6199be07-663c-4792-afaa-ca98a12c9902-kube-api-access-r4q4t\") pod \"lws-controller-manager-7659478ddc-zzqfx\" (UID: \"6199be07-663c-4792-afaa-ca98a12c9902\") " pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.359672 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.359502 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6199be07-663c-4792-afaa-ca98a12c9902-cert\") pod \"lws-controller-manager-7659478ddc-zzqfx\" (UID: \"6199be07-663c-4792-afaa-ca98a12c9902\") " pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.359672 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.359526 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6199be07-663c-4792-afaa-ca98a12c9902-manager-config\") pod \"lws-controller-manager-7659478ddc-zzqfx\" (UID: \"6199be07-663c-4792-afaa-ca98a12c9902\") " pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.359672 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.359543 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6199be07-663c-4792-afaa-ca98a12c9902-metrics-cert\") pod \"lws-controller-manager-7659478ddc-zzqfx\" (UID: \"6199be07-663c-4792-afaa-ca98a12c9902\") " pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.360260 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.360231 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6199be07-663c-4792-afaa-ca98a12c9902-manager-config\") pod \"lws-controller-manager-7659478ddc-zzqfx\" (UID: \"6199be07-663c-4792-afaa-ca98a12c9902\") " pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.362109 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.362081 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6199be07-663c-4792-afaa-ca98a12c9902-cert\") pod \"lws-controller-manager-7659478ddc-zzqfx\" (UID: \"6199be07-663c-4792-afaa-ca98a12c9902\") " pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.362208 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.362154 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6199be07-663c-4792-afaa-ca98a12c9902-metrics-cert\") pod \"lws-controller-manager-7659478ddc-zzqfx\" (UID: \"6199be07-663c-4792-afaa-ca98a12c9902\") " pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.367812 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.367784 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4q4t\" (UniqueName: \"kubernetes.io/projected/6199be07-663c-4792-afaa-ca98a12c9902-kube-api-access-r4q4t\") pod \"lws-controller-manager-7659478ddc-zzqfx\" (UID: \"6199be07-663c-4792-afaa-ca98a12c9902\") " pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.544192 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.544102 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:33.683951 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:33.683926 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx"] Jul 10 17:47:33.686560 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:47:33.686535 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6199be07_663c_4792_afaa_ca98a12c9902.slice/crio-0babc1ffddc17a97b41440c195edc54ffa8ae26e67eb610d8a84234d901b9281 WatchSource:0}: Error finding container 0babc1ffddc17a97b41440c195edc54ffa8ae26e67eb610d8a84234d901b9281: Status 404 returned error can't find the container with id 0babc1ffddc17a97b41440c195edc54ffa8ae26e67eb610d8a84234d901b9281 Jul 10 17:47:34.260180 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:34.260142 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" event={"ID":"6199be07-663c-4792-afaa-ca98a12c9902","Type":"ContainerStarted","Data":"0babc1ffddc17a97b41440c195edc54ffa8ae26e67eb610d8a84234d901b9281"} Jul 10 17:47:36.271449 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:36.271411 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" event={"ID":"6199be07-663c-4792-afaa-ca98a12c9902","Type":"ContainerStarted","Data":"1e5b74b93077fc73aa8e94ce869e9ecaf5c5609133e4c48abfc4714f0e6ec333"} Jul 10 17:47:36.271840 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:36.271523 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:36.290444 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:36.290397 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" podStartSLOduration=1.621767335 podStartE2EDuration="3.290382974s" totalImagesPullingTime="1.668615639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:47:36.28926628 +0000 UTC m=+505.563592974" watchObservedRunningTime="2026-07-10 17:47:36.290382974 +0000 UTC m=+505.564709669" Jul 10 17:47:43.126674 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.126638 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm"] Jul 10 17:47:43.130387 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.130363 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.133416 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.133385 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 17:47:43.133565 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.133515 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 17:47:43.133650 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.133636 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7wd7s\"" Jul 10 17:47:43.133701 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.133691 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 17:47:43.133989 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.133961 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 17:47:43.150190 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.150155 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm"] Jul 10 17:47:43.238111 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.238070 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/250ddbcb-9cbd-469c-9df3-de593b7b736a-manifests\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.238111 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.238109 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq2n8\" (UniqueName: \"kubernetes.io/projected/250ddbcb-9cbd-469c-9df3-de593b7b736a-kube-api-access-fq2n8\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.238504 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.238152 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/250ddbcb-9cbd-469c-9df3-de593b7b736a-tmp\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.238504 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.238234 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/250ddbcb-9cbd-469c-9df3-de593b7b736a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.238504 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.238267 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/250ddbcb-9cbd-469c-9df3-de593b7b736a-webhook-cert\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.339080 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.339046 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/250ddbcb-9cbd-469c-9df3-de593b7b736a-manifests\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.339080 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.339085 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq2n8\" (UniqueName: \"kubernetes.io/projected/250ddbcb-9cbd-469c-9df3-de593b7b736a-kube-api-access-fq2n8\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.339347 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.339126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/250ddbcb-9cbd-469c-9df3-de593b7b736a-tmp\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.339347 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.339159 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/250ddbcb-9cbd-469c-9df3-de593b7b736a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.339347 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.339179 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/250ddbcb-9cbd-469c-9df3-de593b7b736a-webhook-cert\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.339505 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.339484 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/250ddbcb-9cbd-469c-9df3-de593b7b736a-manifests\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.341670 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.341637 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/250ddbcb-9cbd-469c-9df3-de593b7b736a-tmp\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.341787 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.341709 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/250ddbcb-9cbd-469c-9df3-de593b7b736a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.341787 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.341741 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/250ddbcb-9cbd-469c-9df3-de593b7b736a-webhook-cert\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.349096 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.349069 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq2n8\" (UniqueName: \"kubernetes.io/projected/250ddbcb-9cbd-469c-9df3-de593b7b736a-kube-api-access-fq2n8\") pod \"opendatahub-operator-controller-manager-dfdbc665d-n7dsm\" (UID: \"250ddbcb-9cbd-469c-9df3-de593b7b736a\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.441793 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.441703 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:43.586414 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:43.586382 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm"] Jul 10 17:47:43.589202 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:47:43.589171 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250ddbcb_9cbd_469c_9df3_de593b7b736a.slice/crio-f3947324b4305cdedb8b007cd3ca0317556806ff451d636fbdc3649be027aa63 WatchSource:0}: Error finding container f3947324b4305cdedb8b007cd3ca0317556806ff451d636fbdc3649be027aa63: Status 404 returned error can't find the container with id f3947324b4305cdedb8b007cd3ca0317556806ff451d636fbdc3649be027aa63 Jul 10 17:47:44.308068 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:44.308028 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" event={"ID":"250ddbcb-9cbd-469c-9df3-de593b7b736a","Type":"ContainerStarted","Data":"f3947324b4305cdedb8b007cd3ca0317556806ff451d636fbdc3649be027aa63"} Jul 10 17:47:47.278376 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:47.278348 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7659478ddc-zzqfx" Jul 10 17:47:47.326500 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:47.326467 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" event={"ID":"250ddbcb-9cbd-469c-9df3-de593b7b736a","Type":"ContainerStarted","Data":"2fad3618a3ba84a6b5f9b39f0c8f5a579a20ae34a06d841405310909ab00f48a"} Jul 10 17:47:48.333138 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:48.333098 2556 generic.go:356] "Generic (PLEG): container finished" podID="250ddbcb-9cbd-469c-9df3-de593b7b736a" containerID="2fad3618a3ba84a6b5f9b39f0c8f5a579a20ae34a06d841405310909ab00f48a" exitCode=0 Jul 10 17:47:48.333517 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:48.333190 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" event={"ID":"250ddbcb-9cbd-469c-9df3-de593b7b736a","Type":"ContainerDied","Data":"2fad3618a3ba84a6b5f9b39f0c8f5a579a20ae34a06d841405310909ab00f48a"} Jul 10 17:47:49.340274 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:49.340238 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" event={"ID":"250ddbcb-9cbd-469c-9df3-de593b7b736a","Type":"ContainerStarted","Data":"41cf19557c631819a6c4f00a8e5d895d6cbaa555ffeee8d7ee3e881a2921407b"} Jul 10 17:47:49.340734 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:49.340365 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:47:49.362157 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:47:49.362109 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" podStartSLOduration=1.883265709 podStartE2EDuration="6.362091294s" totalImagesPullingTime="3.30547711s" totalInitContainerRuntime="1.173348475s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:47:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:47:49.360320422 +0000 UTC m=+518.634647116" watchObservedRunningTime="2026-07-10 17:47:49.362091294 +0000 UTC m=+518.636417988" Jul 10 17:48:00.347388 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:00.347350 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-n7dsm" Jul 10 17:48:02.570710 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.570619 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w"] Jul 10 17:48:02.574799 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.574778 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" Jul 10 17:48:02.579436 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.579405 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 17:48:02.579573 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.579542 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-l56jd\"" Jul 10 17:48:02.580253 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.580238 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 17:48:02.580609 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.580594 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 17:48:02.580704 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.580597 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 17:48:02.590091 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.590068 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w"] Jul 10 17:48:02.711121 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.711076 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-667bf\" (UniqueName: \"kubernetes.io/projected/04073a5d-860c-47f4-8f3d-ddfc2477fc62-kube-api-access-667bf\") pod \"kube-auth-proxy-5b544bffd9-89d4w\" (UID: \"04073a5d-860c-47f4-8f3d-ddfc2477fc62\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" Jul 10 17:48:02.711310 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.711135 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04073a5d-860c-47f4-8f3d-ddfc2477fc62-tmp\") pod \"kube-auth-proxy-5b544bffd9-89d4w\" (UID: \"04073a5d-860c-47f4-8f3d-ddfc2477fc62\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" Jul 10 17:48:02.711310 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.711200 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04073a5d-860c-47f4-8f3d-ddfc2477fc62-tls-certs\") pod \"kube-auth-proxy-5b544bffd9-89d4w\" (UID: \"04073a5d-860c-47f4-8f3d-ddfc2477fc62\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" Jul 10 17:48:02.812124 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.812086 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667bf\" (UniqueName: \"kubernetes.io/projected/04073a5d-860c-47f4-8f3d-ddfc2477fc62-kube-api-access-667bf\") pod \"kube-auth-proxy-5b544bffd9-89d4w\" (UID: \"04073a5d-860c-47f4-8f3d-ddfc2477fc62\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" Jul 10 17:48:02.812328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.812134 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04073a5d-860c-47f4-8f3d-ddfc2477fc62-tmp\") pod \"kube-auth-proxy-5b544bffd9-89d4w\" (UID: \"04073a5d-860c-47f4-8f3d-ddfc2477fc62\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" Jul 10 17:48:02.812328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.812157 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04073a5d-860c-47f4-8f3d-ddfc2477fc62-tls-certs\") pod \"kube-auth-proxy-5b544bffd9-89d4w\" (UID: \"04073a5d-860c-47f4-8f3d-ddfc2477fc62\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" Jul 10 17:48:02.814547 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.814523 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04073a5d-860c-47f4-8f3d-ddfc2477fc62-tmp\") pod \"kube-auth-proxy-5b544bffd9-89d4w\" (UID: \"04073a5d-860c-47f4-8f3d-ddfc2477fc62\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" Jul 10 17:48:02.814839 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.814818 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04073a5d-860c-47f4-8f3d-ddfc2477fc62-tls-certs\") pod \"kube-auth-proxy-5b544bffd9-89d4w\" (UID: \"04073a5d-860c-47f4-8f3d-ddfc2477fc62\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" Jul 10 17:48:02.828395 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.828330 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-667bf\" (UniqueName: \"kubernetes.io/projected/04073a5d-860c-47f4-8f3d-ddfc2477fc62-kube-api-access-667bf\") pod \"kube-auth-proxy-5b544bffd9-89d4w\" (UID: \"04073a5d-860c-47f4-8f3d-ddfc2477fc62\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" Jul 10 17:48:02.886189 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:02.886152 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" Jul 10 17:48:03.066107 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:03.066075 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w"] Jul 10 17:48:03.067311 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:48:03.067284 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04073a5d_860c_47f4_8f3d_ddfc2477fc62.slice/crio-81ca8bdb5b9780c9a1c590034ab3e48393c4c1c8cfd102249cd1d970621d86b2 WatchSource:0}: Error finding container 81ca8bdb5b9780c9a1c590034ab3e48393c4c1c8cfd102249cd1d970621d86b2: Status 404 returned error can't find the container with id 81ca8bdb5b9780c9a1c590034ab3e48393c4c1c8cfd102249cd1d970621d86b2 Jul 10 17:48:03.404827 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:03.404792 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" event={"ID":"04073a5d-860c-47f4-8f3d-ddfc2477fc62","Type":"ContainerStarted","Data":"81ca8bdb5b9780c9a1c590034ab3e48393c4c1c8cfd102249cd1d970621d86b2"} Jul 10 17:48:06.421600 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:06.421503 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" event={"ID":"04073a5d-860c-47f4-8f3d-ddfc2477fc62","Type":"ContainerStarted","Data":"607557bd8a87db4b98d3f424bc1eca4189b94fe1f158288f290b03c8c3cc00f9"} Jul 10 17:48:06.437957 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:48:06.437900 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b544bffd9-89d4w" podStartSLOduration=1.349084647 podStartE2EDuration="4.437880707s" totalImagesPullingTime="3.08879606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:48:06.436892055 +0000 UTC m=+535.711218767" watchObservedRunningTime="2026-07-10 17:48:06.437880707 +0000 UTC m=+535.712207443" Jul 10 17:49:11.114242 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:11.114212 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 17:49:11.114739 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:11.114422 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 17:49:22.723489 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.723454 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b557c49f5-8qnm2"] Jul 10 17:49:22.726955 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.726935 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.739808 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.739784 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b557c49f5-8qnm2"] Jul 10 17:49:22.806742 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.806707 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn9vx\" (UniqueName: \"kubernetes.io/projected/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-kube-api-access-xn9vx\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.806742 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.806746 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-trusted-ca-bundle\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.807046 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.806770 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-console-config\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.807046 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.806871 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-console-oauth-config\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.807046 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.806904 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-service-ca\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.807046 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.807005 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-oauth-serving-cert\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.807046 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.807045 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-console-serving-cert\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.908143 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.908102 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn9vx\" (UniqueName: \"kubernetes.io/projected/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-kube-api-access-xn9vx\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.908143 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.908142 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-trusted-ca-bundle\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.908386 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.908165 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-console-config\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.908386 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.908201 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-console-oauth-config\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.908386 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.908218 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-service-ca\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.908386 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.908241 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-oauth-serving-cert\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.908386 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.908266 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-console-serving-cert\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.909000 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.908938 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-console-config\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.909124 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.909018 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-service-ca\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.909124 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.909046 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-oauth-serving-cert\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.909367 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.909345 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-trusted-ca-bundle\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.910809 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.910782 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-console-serving-cert\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.910908 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.910861 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-console-oauth-config\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:22.916105 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:22.916086 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn9vx\" (UniqueName: \"kubernetes.io/projected/0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2-kube-api-access-xn9vx\") pod \"console-7b557c49f5-8qnm2\" (UID: \"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2\") " pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:23.037451 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:23.037353 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:23.176830 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:23.176805 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b557c49f5-8qnm2"] Jul 10 17:49:23.179625 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:49:23.179590 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0a9e34_d2f8_44f7_b8ef_a6c1d37c71e2.slice/crio-cea444ea815cad111d99b75859a047ec81cc575af63520feb336cdbb9e8696a7 WatchSource:0}: Error finding container cea444ea815cad111d99b75859a047ec81cc575af63520feb336cdbb9e8696a7: Status 404 returned error can't find the container with id cea444ea815cad111d99b75859a047ec81cc575af63520feb336cdbb9e8696a7 Jul 10 17:49:23.773318 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:23.773283 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b557c49f5-8qnm2" event={"ID":"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2","Type":"ContainerStarted","Data":"e59cef9161626d5ccb7530a8e43b7bf9d945d4d9a07a35c5bec489bf2dff2231"} Jul 10 17:49:23.773679 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:23.773335 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b557c49f5-8qnm2" event={"ID":"0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2","Type":"ContainerStarted","Data":"cea444ea815cad111d99b75859a047ec81cc575af63520feb336cdbb9e8696a7"} Jul 10 17:49:23.790380 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:23.790332 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b557c49f5-8qnm2" podStartSLOduration=1.790317426 podStartE2EDuration="1.790317426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:49:23.789611283 +0000 UTC m=+613.063937979" watchObservedRunningTime="2026-07-10 17:49:23.790317426 +0000 UTC m=+613.064644122" Jul 10 17:49:33.037814 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:33.037726 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:33.038291 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:33.037838 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:33.042535 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:33.042514 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:33.821690 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:33.821655 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b557c49f5-8qnm2" Jul 10 17:49:33.867791 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:33.867758 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9fb8465f8-sc2fd"] Jul 10 17:49:45.340284 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.340251 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk"] Jul 10 17:49:45.343626 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.343608 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" Jul 10 17:49:45.348496 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.348469 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 17:49:45.348647 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.348517 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 17:49:45.348749 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.348736 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 17:49:45.348749 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.348744 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 17:49:45.349516 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.349500 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n84lr\"" Jul 10 17:49:45.354683 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.354651 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk"] Jul 10 17:49:45.414612 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.414568 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9dvm\" (UniqueName: \"kubernetes.io/projected/24516b57-3607-4080-b795-4ed5e576c8b4-kube-api-access-k9dvm\") pod \"kuadrant-console-plugin-66b5774985-nzzqk\" (UID: \"24516b57-3607-4080-b795-4ed5e576c8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" Jul 10 17:49:45.414861 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.414673 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24516b57-3607-4080-b795-4ed5e576c8b4-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nzzqk\" (UID: \"24516b57-3607-4080-b795-4ed5e576c8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" Jul 10 17:49:45.414861 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.414728 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/24516b57-3607-4080-b795-4ed5e576c8b4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nzzqk\" (UID: \"24516b57-3607-4080-b795-4ed5e576c8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" Jul 10 17:49:45.515318 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.515277 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dvm\" (UniqueName: \"kubernetes.io/projected/24516b57-3607-4080-b795-4ed5e576c8b4-kube-api-access-k9dvm\") pod \"kuadrant-console-plugin-66b5774985-nzzqk\" (UID: \"24516b57-3607-4080-b795-4ed5e576c8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" Jul 10 17:49:45.515527 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.515389 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24516b57-3607-4080-b795-4ed5e576c8b4-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nzzqk\" (UID: \"24516b57-3607-4080-b795-4ed5e576c8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" Jul 10 17:49:45.515527 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.515439 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/24516b57-3607-4080-b795-4ed5e576c8b4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nzzqk\" (UID: \"24516b57-3607-4080-b795-4ed5e576c8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" Jul 10 17:49:45.516117 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.516095 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24516b57-3607-4080-b795-4ed5e576c8b4-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nzzqk\" (UID: \"24516b57-3607-4080-b795-4ed5e576c8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" Jul 10 17:49:45.518270 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.518237 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/24516b57-3607-4080-b795-4ed5e576c8b4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nzzqk\" (UID: \"24516b57-3607-4080-b795-4ed5e576c8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" Jul 10 17:49:45.531390 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.531355 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dvm\" (UniqueName: \"kubernetes.io/projected/24516b57-3607-4080-b795-4ed5e576c8b4-kube-api-access-k9dvm\") pod \"kuadrant-console-plugin-66b5774985-nzzqk\" (UID: \"24516b57-3607-4080-b795-4ed5e576c8b4\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" Jul 10 17:49:45.654254 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.654215 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" Jul 10 17:49:45.795851 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.795827 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk"] Jul 10 17:49:45.798380 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:49:45.798345 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24516b57_3607_4080_b795_4ed5e576c8b4.slice/crio-b578d0661aee242cdc8db59a3e04faa716b5085ef2706bb8c9ce294892219809 WatchSource:0}: Error finding container b578d0661aee242cdc8db59a3e04faa716b5085ef2706bb8c9ce294892219809: Status 404 returned error can't find the container with id b578d0661aee242cdc8db59a3e04faa716b5085ef2706bb8c9ce294892219809 Jul 10 17:49:45.874035 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:45.873997 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" event={"ID":"24516b57-3607-4080-b795-4ed5e576c8b4","Type":"ContainerStarted","Data":"b578d0661aee242cdc8db59a3e04faa716b5085ef2706bb8c9ce294892219809"} Jul 10 17:49:58.896524 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:49:58.896482 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9fb8465f8-sc2fd" podUID="b5046f8d-71be-4389-b77f-0c4482cbc6b1" containerName="console" containerID="cri-o://69220a0f54c0f1b32629d547781c5ffebdfec03e3335e7506b259021b7919d05" gracePeriod=15 Jul 10 17:50:00.750390 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:00.750347 2556 patch_prober.go:28] interesting pod/console-9fb8465f8-sc2fd container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.24:8443/health\": dial tcp 10.134.0.24:8443: connect: connection refused" start-of-body= Jul 10 17:50:00.750812 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:00.750429 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-9fb8465f8-sc2fd" podUID="b5046f8d-71be-4389-b77f-0c4482cbc6b1" containerName="console" probeResult="failure" output="Get \"https://10.134.0.24:8443/health\": dial tcp 10.134.0.24:8443: connect: connection refused" Jul 10 17:50:09.625987 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.625944 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:50:09.643845 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.643822 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj75r\" (UniqueName: \"kubernetes.io/projected/b5046f8d-71be-4389-b77f-0c4482cbc6b1-kube-api-access-tj75r\") pod \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " Jul 10 17:50:09.643985 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.643878 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-service-ca\") pod \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " Jul 10 17:50:09.643985 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.643894 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-oauth-serving-cert\") pod \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " Jul 10 17:50:09.643985 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.643920 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-trusted-ca-bundle\") pod \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " Jul 10 17:50:09.643985 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.643946 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-config\") pod \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " Jul 10 17:50:09.644263 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.644067 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-serving-cert\") pod \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " Jul 10 17:50:09.644263 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.644115 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-oauth-config\") pod \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\" (UID: \"b5046f8d-71be-4389-b77f-0c4482cbc6b1\") " Jul 10 17:50:09.644436 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.644394 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-service-ca" (OuterVolumeSpecName: "service-ca") pod "b5046f8d-71be-4389-b77f-0c4482cbc6b1" (UID: "b5046f8d-71be-4389-b77f-0c4482cbc6b1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:50:09.644436 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.644406 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-config" (OuterVolumeSpecName: "console-config") pod "b5046f8d-71be-4389-b77f-0c4482cbc6b1" (UID: "b5046f8d-71be-4389-b77f-0c4482cbc6b1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:50:09.644436 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.644412 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b5046f8d-71be-4389-b77f-0c4482cbc6b1" (UID: "b5046f8d-71be-4389-b77f-0c4482cbc6b1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:50:09.645098 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.645063 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b5046f8d-71be-4389-b77f-0c4482cbc6b1" (UID: "b5046f8d-71be-4389-b77f-0c4482cbc6b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:50:09.646683 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.646649 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5046f8d-71be-4389-b77f-0c4482cbc6b1-kube-api-access-tj75r" (OuterVolumeSpecName: "kube-api-access-tj75r") pod "b5046f8d-71be-4389-b77f-0c4482cbc6b1" (UID: "b5046f8d-71be-4389-b77f-0c4482cbc6b1"). InnerVolumeSpecName "kube-api-access-tj75r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:50:09.646917 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.646890 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b5046f8d-71be-4389-b77f-0c4482cbc6b1" (UID: "b5046f8d-71be-4389-b77f-0c4482cbc6b1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:50:09.647545 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.647520 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b5046f8d-71be-4389-b77f-0c4482cbc6b1" (UID: "b5046f8d-71be-4389-b77f-0c4482cbc6b1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:50:09.745232 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.745139 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-oauth-config\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:50:09.745232 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.745171 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj75r\" (UniqueName: \"kubernetes.io/projected/b5046f8d-71be-4389-b77f-0c4482cbc6b1-kube-api-access-tj75r\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:50:09.745232 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.745182 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-service-ca\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:50:09.745232 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.745190 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-oauth-serving-cert\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:50:09.745232 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.745200 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-trusted-ca-bundle\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:50:09.745232 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.745208 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-config\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:50:09.745232 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:09.745216 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5046f8d-71be-4389-b77f-0c4482cbc6b1-console-serving-cert\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:50:10.004497 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:10.004401 2556 generic.go:356] "Generic (PLEG): container finished" podID="b5046f8d-71be-4389-b77f-0c4482cbc6b1" containerID="69220a0f54c0f1b32629d547781c5ffebdfec03e3335e7506b259021b7919d05" exitCode=0 Jul 10 17:50:10.004497 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:10.004453 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fb8465f8-sc2fd" Jul 10 17:50:10.004497 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:10.004491 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fb8465f8-sc2fd" event={"ID":"b5046f8d-71be-4389-b77f-0c4482cbc6b1","Type":"ContainerDied","Data":"69220a0f54c0f1b32629d547781c5ffebdfec03e3335e7506b259021b7919d05"} Jul 10 17:50:10.005027 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:10.004544 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fb8465f8-sc2fd" event={"ID":"b5046f8d-71be-4389-b77f-0c4482cbc6b1","Type":"ContainerDied","Data":"d45671fc038b9d08c3545287ac5193f0ec86c5a1eff11f3a6de46af2c5bb4040"} Jul 10 17:50:10.005027 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:10.004572 2556 scope.go:117] "RemoveContainer" containerID="69220a0f54c0f1b32629d547781c5ffebdfec03e3335e7506b259021b7919d05" Jul 10 17:50:10.006316 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:10.006291 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" event={"ID":"24516b57-3607-4080-b795-4ed5e576c8b4","Type":"ContainerStarted","Data":"ff69e8171422a4a9f497c6253d033a0d02ccdaeca9d1f0441baf0e327b0d74fa"} Jul 10 17:50:10.013805 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:10.013786 2556 scope.go:117] "RemoveContainer" containerID="69220a0f54c0f1b32629d547781c5ffebdfec03e3335e7506b259021b7919d05" Jul 10 17:50:10.014108 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:50:10.014090 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69220a0f54c0f1b32629d547781c5ffebdfec03e3335e7506b259021b7919d05\": container with ID starting with 69220a0f54c0f1b32629d547781c5ffebdfec03e3335e7506b259021b7919d05 not found: ID does not exist" containerID="69220a0f54c0f1b32629d547781c5ffebdfec03e3335e7506b259021b7919d05" Jul 10 17:50:10.014167 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:10.014117 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69220a0f54c0f1b32629d547781c5ffebdfec03e3335e7506b259021b7919d05"} err="failed to get container status \"69220a0f54c0f1b32629d547781c5ffebdfec03e3335e7506b259021b7919d05\": rpc error: code = NotFound desc = could not find container \"69220a0f54c0f1b32629d547781c5ffebdfec03e3335e7506b259021b7919d05\": container with ID starting with 69220a0f54c0f1b32629d547781c5ffebdfec03e3335e7506b259021b7919d05 not found: ID does not exist" Jul 10 17:50:10.027280 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:10.026030 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzzqk" podStartSLOduration=1.276785204 podStartE2EDuration="25.026007365s" totalImagesPullingTime="23.749222161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:50:10.022642995 +0000 UTC m=+659.296969691" watchObservedRunningTime="2026-07-10 17:50:10.026007365 +0000 UTC m=+659.300334061" Jul 10 17:50:10.043316 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:10.043277 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9fb8465f8-sc2fd"] Jul 10 17:50:10.048393 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:10.048364 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9fb8465f8-sc2fd"] Jul 10 17:50:11.255124 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:11.255089 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5046f8d-71be-4389-b77f-0c4482cbc6b1" path="/var/lib/kubelet/pods/b5046f8d-71be-4389-b77f-0c4482cbc6b1/volumes" Jul 10 17:50:26.551644 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.551612 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:50:26.552132 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.551961 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5046f8d-71be-4389-b77f-0c4482cbc6b1" containerName="console" Jul 10 17:50:26.552132 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.551987 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5046f8d-71be-4389-b77f-0c4482cbc6b1" containerName="console" Jul 10 17:50:26.552132 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.552043 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5046f8d-71be-4389-b77f-0c4482cbc6b1" containerName="console" Jul 10 17:50:26.586880 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.586845 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:50:26.586880 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.586887 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:50:26.587094 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.586999 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-99mbl" Jul 10 17:50:26.589629 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.589608 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 17:50:26.693222 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.693181 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bwqg\" (UniqueName: \"kubernetes.io/projected/77995e24-aa95-4636-9117-e764fe6d390f-kube-api-access-6bwqg\") pod \"limitador-limitador-54d6d988d5-99mbl\" (UID: \"77995e24-aa95-4636-9117-e764fe6d390f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-99mbl" Jul 10 17:50:26.693398 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.693311 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77995e24-aa95-4636-9117-e764fe6d390f-config-file\") pod \"limitador-limitador-54d6d988d5-99mbl\" (UID: \"77995e24-aa95-4636-9117-e764fe6d390f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-99mbl" Jul 10 17:50:26.794689 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.794647 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77995e24-aa95-4636-9117-e764fe6d390f-config-file\") pod \"limitador-limitador-54d6d988d5-99mbl\" (UID: \"77995e24-aa95-4636-9117-e764fe6d390f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-99mbl" Jul 10 17:50:26.794825 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.794701 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bwqg\" (UniqueName: \"kubernetes.io/projected/77995e24-aa95-4636-9117-e764fe6d390f-kube-api-access-6bwqg\") pod \"limitador-limitador-54d6d988d5-99mbl\" (UID: \"77995e24-aa95-4636-9117-e764fe6d390f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-99mbl" Jul 10 17:50:26.795335 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.795316 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77995e24-aa95-4636-9117-e764fe6d390f-config-file\") pod \"limitador-limitador-54d6d988d5-99mbl\" (UID: \"77995e24-aa95-4636-9117-e764fe6d390f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-99mbl" Jul 10 17:50:26.804479 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.804410 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bwqg\" (UniqueName: \"kubernetes.io/projected/77995e24-aa95-4636-9117-e764fe6d390f-kube-api-access-6bwqg\") pod \"limitador-limitador-54d6d988d5-99mbl\" (UID: \"77995e24-aa95-4636-9117-e764fe6d390f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-99mbl" Jul 10 17:50:26.897383 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:26.897343 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-99mbl" Jul 10 17:50:27.033473 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:27.033410 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:50:27.036022 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:50:27.035988 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77995e24_aa95_4636_9117_e764fe6d390f.slice/crio-26487c2a568f9000c0c28d581ab828e331f71ec9044c4686b00fdb2a8a60c212 WatchSource:0}: Error finding container 26487c2a568f9000c0c28d581ab828e331f71ec9044c4686b00fdb2a8a60c212: Status 404 returned error can't find the container with id 26487c2a568f9000c0c28d581ab828e331f71ec9044c4686b00fdb2a8a60c212 Jul 10 17:50:27.043225 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:27.043202 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-lxxgm"] Jul 10 17:50:27.046425 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:27.046408 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lxxgm" Jul 10 17:50:27.048841 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:27.048818 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2xs9m\"" Jul 10 17:50:27.055184 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:27.055125 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lxxgm"] Jul 10 17:50:27.093913 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:27.093880 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-99mbl" event={"ID":"77995e24-aa95-4636-9117-e764fe6d390f","Type":"ContainerStarted","Data":"26487c2a568f9000c0c28d581ab828e331f71ec9044c4686b00fdb2a8a60c212"} Jul 10 17:50:27.098314 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:27.098282 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2whc\" (UniqueName: \"kubernetes.io/projected/cbd15def-e27b-43aa-bafd-9d71338d055a-kube-api-access-h2whc\") pod \"authorino-75d556bd77-lxxgm\" (UID: \"cbd15def-e27b-43aa-bafd-9d71338d055a\") " pod="kuadrant-system/authorino-75d556bd77-lxxgm" Jul 10 17:50:27.199405 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:27.199358 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2whc\" (UniqueName: \"kubernetes.io/projected/cbd15def-e27b-43aa-bafd-9d71338d055a-kube-api-access-h2whc\") pod \"authorino-75d556bd77-lxxgm\" (UID: \"cbd15def-e27b-43aa-bafd-9d71338d055a\") " pod="kuadrant-system/authorino-75d556bd77-lxxgm" Jul 10 17:50:27.207186 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:27.207153 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2whc\" (UniqueName: \"kubernetes.io/projected/cbd15def-e27b-43aa-bafd-9d71338d055a-kube-api-access-h2whc\") pod \"authorino-75d556bd77-lxxgm\" (UID: \"cbd15def-e27b-43aa-bafd-9d71338d055a\") " pod="kuadrant-system/authorino-75d556bd77-lxxgm" Jul 10 17:50:27.356664 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:27.356631 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lxxgm" Jul 10 17:50:27.534794 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:27.534751 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lxxgm"] Jul 10 17:50:27.543261 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:50:27.543214 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbd15def_e27b_43aa_bafd_9d71338d055a.slice/crio-318b346dffca9c78c8beb6f3b0411a5f208699c6fde009bd6edfe201cc2366bd WatchSource:0}: Error finding container 318b346dffca9c78c8beb6f3b0411a5f208699c6fde009bd6edfe201cc2366bd: Status 404 returned error can't find the container with id 318b346dffca9c78c8beb6f3b0411a5f208699c6fde009bd6edfe201cc2366bd Jul 10 17:50:28.102130 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:28.102087 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lxxgm" event={"ID":"cbd15def-e27b-43aa-bafd-9d71338d055a","Type":"ContainerStarted","Data":"318b346dffca9c78c8beb6f3b0411a5f208699c6fde009bd6edfe201cc2366bd"} Jul 10 17:50:32.125455 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:32.125421 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-99mbl" event={"ID":"77995e24-aa95-4636-9117-e764fe6d390f","Type":"ContainerStarted","Data":"65a1164ee028df24232c37f325d0be5e61601556f2a591732534b4a9849cc9d4"} Jul 10 17:50:32.125909 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:32.125499 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-99mbl" Jul 10 17:50:32.126832 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:32.126811 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lxxgm" event={"ID":"cbd15def-e27b-43aa-bafd-9d71338d055a","Type":"ContainerStarted","Data":"939da7bf0d56932fb5cf75b48f53e770cff925791e599fb9ee55dbe192ed8733"} Jul 10 17:50:32.141950 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:32.141906 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-99mbl" podStartSLOduration=1.8525965709999999 podStartE2EDuration="6.141888168s" totalImagesPullingTime="4.289291597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:50:32.141423403 +0000 UTC m=+681.415750098" watchObservedRunningTime="2026-07-10 17:50:32.141888168 +0000 UTC m=+681.416214885" Jul 10 17:50:32.156497 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:32.156446 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-lxxgm" podStartSLOduration=1.367334255 podStartE2EDuration="5.156425349s" totalImagesPullingTime="3.789091094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:50:32.155308641 +0000 UTC m=+681.429635338" watchObservedRunningTime="2026-07-10 17:50:32.156425349 +0000 UTC m=+681.430752043" Jul 10 17:50:43.138819 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:50:43.138791 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-99mbl" Jul 10 17:51:02.279049 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:02.278952 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lxxgm"] Jul 10 17:51:02.279440 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:02.279163 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-lxxgm" podUID="cbd15def-e27b-43aa-bafd-9d71338d055a" containerName="authorino" containerID="cri-o://939da7bf0d56932fb5cf75b48f53e770cff925791e599fb9ee55dbe192ed8733" gracePeriod=30 Jul 10 17:51:02.532212 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:02.532152 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lxxgm" Jul 10 17:51:02.605554 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:02.605512 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2whc\" (UniqueName: \"kubernetes.io/projected/cbd15def-e27b-43aa-bafd-9d71338d055a-kube-api-access-h2whc\") pod \"cbd15def-e27b-43aa-bafd-9d71338d055a\" (UID: \"cbd15def-e27b-43aa-bafd-9d71338d055a\") " Jul 10 17:51:02.607848 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:02.607810 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd15def-e27b-43aa-bafd-9d71338d055a-kube-api-access-h2whc" (OuterVolumeSpecName: "kube-api-access-h2whc") pod "cbd15def-e27b-43aa-bafd-9d71338d055a" (UID: "cbd15def-e27b-43aa-bafd-9d71338d055a"). InnerVolumeSpecName "kube-api-access-h2whc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:51:02.706593 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:02.706551 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2whc\" (UniqueName: \"kubernetes.io/projected/cbd15def-e27b-43aa-bafd-9d71338d055a-kube-api-access-h2whc\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 17:51:03.273713 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:03.273684 2556 generic.go:356] "Generic (PLEG): container finished" podID="cbd15def-e27b-43aa-bafd-9d71338d055a" containerID="939da7bf0d56932fb5cf75b48f53e770cff925791e599fb9ee55dbe192ed8733" exitCode=0 Jul 10 17:51:03.273864 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:03.273725 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lxxgm" Jul 10 17:51:03.273864 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:03.273782 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lxxgm" event={"ID":"cbd15def-e27b-43aa-bafd-9d71338d055a","Type":"ContainerDied","Data":"939da7bf0d56932fb5cf75b48f53e770cff925791e599fb9ee55dbe192ed8733"} Jul 10 17:51:03.273864 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:03.273810 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lxxgm" event={"ID":"cbd15def-e27b-43aa-bafd-9d71338d055a","Type":"ContainerDied","Data":"318b346dffca9c78c8beb6f3b0411a5f208699c6fde009bd6edfe201cc2366bd"} Jul 10 17:51:03.273864 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:03.273827 2556 scope.go:117] "RemoveContainer" containerID="939da7bf0d56932fb5cf75b48f53e770cff925791e599fb9ee55dbe192ed8733" Jul 10 17:51:03.282790 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:03.282604 2556 scope.go:117] "RemoveContainer" containerID="939da7bf0d56932fb5cf75b48f53e770cff925791e599fb9ee55dbe192ed8733" Jul 10 17:51:03.283062 ip-10-0-137-103 kubenswrapper[2556]: E0710 17:51:03.282878 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939da7bf0d56932fb5cf75b48f53e770cff925791e599fb9ee55dbe192ed8733\": container with ID starting with 939da7bf0d56932fb5cf75b48f53e770cff925791e599fb9ee55dbe192ed8733 not found: ID does not exist" containerID="939da7bf0d56932fb5cf75b48f53e770cff925791e599fb9ee55dbe192ed8733" Jul 10 17:51:03.283062 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:03.282903 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939da7bf0d56932fb5cf75b48f53e770cff925791e599fb9ee55dbe192ed8733"} err="failed to get container status \"939da7bf0d56932fb5cf75b48f53e770cff925791e599fb9ee55dbe192ed8733\": rpc error: code = NotFound desc = could not find container \"939da7bf0d56932fb5cf75b48f53e770cff925791e599fb9ee55dbe192ed8733\": container with ID starting with 939da7bf0d56932fb5cf75b48f53e770cff925791e599fb9ee55dbe192ed8733 not found: ID does not exist" Jul 10 17:51:03.291505 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:03.291482 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lxxgm"] Jul 10 17:51:03.294889 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:03.294869 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lxxgm"] Jul 10 17:51:05.254882 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:05.254848 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbd15def-e27b-43aa-bafd-9d71338d055a" path="/var/lib/kubelet/pods/cbd15def-e27b-43aa-bafd-9d71338d055a/volumes" Jul 10 17:51:43.888532 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:51:43.888493 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:52:00.656415 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:00.656383 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:52:08.755315 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:08.755276 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:52:17.456672 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:17.456627 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:52:23.255048 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.255022 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn"] Jul 10 17:52:23.255415 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.255339 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbd15def-e27b-43aa-bafd-9d71338d055a" containerName="authorino" Jul 10 17:52:23.255415 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.255358 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbd15def-e27b-43aa-bafd-9d71338d055a" containerName="authorino" Jul 10 17:52:23.255486 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.255435 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbd15def-e27b-43aa-bafd-9d71338d055a" containerName="authorino" Jul 10 17:52:23.257250 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.257230 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.260124 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.260096 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zxrkb\"" Jul 10 17:52:23.260917 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.260882 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 17:52:23.261054 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.260929 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 17:52:23.261054 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.261003 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 17:52:23.261943 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.261921 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn"] Jul 10 17:52:23.423741 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.423707 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ff4e515-56b8-470c-b9c5-f734b5c83279-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.423922 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.423754 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fl4p\" (UniqueName: \"kubernetes.io/projected/4ff4e515-56b8-470c-b9c5-f734b5c83279-kube-api-access-9fl4p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.423922 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.423789 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ff4e515-56b8-470c-b9c5-f734b5c83279-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.423922 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.423850 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ff4e515-56b8-470c-b9c5-f734b5c83279-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.423922 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.423905 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ff4e515-56b8-470c-b9c5-f734b5c83279-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.424114 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.423931 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ff4e515-56b8-470c-b9c5-f734b5c83279-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.524933 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.524832 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ff4e515-56b8-470c-b9c5-f734b5c83279-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.524933 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.524912 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ff4e515-56b8-470c-b9c5-f734b5c83279-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.525201 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.524956 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fl4p\" (UniqueName: \"kubernetes.io/projected/4ff4e515-56b8-470c-b9c5-f734b5c83279-kube-api-access-9fl4p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.525201 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.525029 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ff4e515-56b8-470c-b9c5-f734b5c83279-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.525201 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.525058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ff4e515-56b8-470c-b9c5-f734b5c83279-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.525359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.525221 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ff4e515-56b8-470c-b9c5-f734b5c83279-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.525359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.525270 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ff4e515-56b8-470c-b9c5-f734b5c83279-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.525359 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.525231 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ff4e515-56b8-470c-b9c5-f734b5c83279-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.525505 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.525444 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ff4e515-56b8-470c-b9c5-f734b5c83279-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.527420 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.527392 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ff4e515-56b8-470c-b9c5-f734b5c83279-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.527676 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.527655 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ff4e515-56b8-470c-b9c5-f734b5c83279-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.532830 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.532808 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fl4p\" (UniqueName: \"kubernetes.io/projected/4ff4e515-56b8-470c-b9c5-f734b5c83279-kube-api-access-9fl4p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn\" (UID: \"4ff4e515-56b8-470c-b9c5-f734b5c83279\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.569105 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.569059 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:23.707743 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.707717 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn"] Jul 10 17:52:23.710449 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:52:23.710417 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff4e515_56b8_470c_b9c5_f734b5c83279.slice/crio-0bc6fcb5f7a4cf9968495f0a2564fb76819e15c2f597e970ae1824a555538102 WatchSource:0}: Error finding container 0bc6fcb5f7a4cf9968495f0a2564fb76819e15c2f597e970ae1824a555538102: Status 404 returned error can't find the container with id 0bc6fcb5f7a4cf9968495f0a2564fb76819e15c2f597e970ae1824a555538102 Jul 10 17:52:23.711712 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:23.711694 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:52:24.653244 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:24.653213 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" event={"ID":"4ff4e515-56b8-470c-b9c5-f734b5c83279","Type":"ContainerStarted","Data":"0bc6fcb5f7a4cf9968495f0a2564fb76819e15c2f597e970ae1824a555538102"} Jul 10 17:52:24.956335 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:24.956248 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:52:25.662409 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:25.662371 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" event={"ID":"4ff4e515-56b8-470c-b9c5-f734b5c83279","Type":"ContainerStarted","Data":"ffdab18567e5c1d952f9dc2e3936e1e9b8b4d742dbbb41d84c5a6608a276002f"} Jul 10 17:52:25.679948 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:25.679897 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" podStartSLOduration=1.706703117 podStartE2EDuration="2.679879893s" totalImagesPullingTime="973.176776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:52:25.678223549 +0000 UTC m=+794.952550243" watchObservedRunningTime="2026-07-10 17:52:25.679879893 +0000 UTC m=+794.954206566" Jul 10 17:52:30.656417 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:30.656375 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:52:31.766328 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.766246 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw"] Jul 10 17:52:31.769026 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.769005 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.771527 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.771509 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 10 17:52:31.778381 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.778359 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw"] Jul 10 17:52:31.800665 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.800632 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.800832 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.800675 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.800832 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.800699 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.800832 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.800756 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.800832 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.800786 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.800963 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.800835 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbdxm\" (UniqueName: \"kubernetes.io/projected/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-kube-api-access-bbdxm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.902302 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.902266 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbdxm\" (UniqueName: \"kubernetes.io/projected/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-kube-api-access-bbdxm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.902461 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.902331 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.902461 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.902363 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.902461 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.902392 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.902461 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.902428 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.902461 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.902451 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.902795 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.902774 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.902861 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.902824 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.902913 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.902859 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.904906 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.904881 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.905238 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.905219 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:31.912790 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:31.912749 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbdxm\" (UniqueName: \"kubernetes.io/projected/151fabd2-9670-4a3c-b8c5-62a60e6c0ead-kube-api-access-bbdxm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw\" (UID: \"151fabd2-9670-4a3c-b8c5-62a60e6c0ead\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:32.078288 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:32.078202 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:32.230043 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:32.230013 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw"] Jul 10 17:52:32.231102 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:52:32.231076 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod151fabd2_9670_4a3c_b8c5_62a60e6c0ead.slice/crio-1f250d02a49f32544502b78b889437814642e573628248493a076f2a405a12f4 WatchSource:0}: Error finding container 1f250d02a49f32544502b78b889437814642e573628248493a076f2a405a12f4: Status 404 returned error can't find the container with id 1f250d02a49f32544502b78b889437814642e573628248493a076f2a405a12f4 Jul 10 17:52:32.697366 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:32.697275 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" event={"ID":"151fabd2-9670-4a3c-b8c5-62a60e6c0ead","Type":"ContainerStarted","Data":"abf2e79fdc3d3941dc2c437bc571ecab91df20591d0dc3410e75b1186943c07b"} Jul 10 17:52:32.697366 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:32.697314 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" event={"ID":"151fabd2-9670-4a3c-b8c5-62a60e6c0ead","Type":"ContainerStarted","Data":"1f250d02a49f32544502b78b889437814642e573628248493a076f2a405a12f4"} Jul 10 17:52:32.715856 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:32.715800 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" podStartSLOduration=1.5405132030000002 podStartE2EDuration="1.715782244s" totalImagesPullingTime="175.269041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:52:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:52:32.713676784 +0000 UTC m=+801.988003497" watchObservedRunningTime="2026-07-10 17:52:32.715782244 +0000 UTC m=+801.990108939" Jul 10 17:52:33.570112 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:33.570059 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:33.570611 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:33.570147 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:33.582631 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:33.582603 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:33.713352 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:33.713325 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn" Jul 10 17:52:39.855050 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:39.855014 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:52:42.078757 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:42.078715 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:42.078757 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:42.078759 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:42.091421 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:42.091396 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:52:42.758940 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:52:42.758905 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw" Jul 10 17:53:28.195808 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.195766 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f55b4dd8-2qz9c"] Jul 10 17:53:28.198185 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.198166 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f55b4dd8-2qz9c" Jul 10 17:53:28.201729 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.201703 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2xs9m\"" Jul 10 17:53:28.201866 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.201707 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 17:53:28.210511 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.210485 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f55b4dd8-2qz9c"] Jul 10 17:53:28.297595 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.297552 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvxw\" (UniqueName: \"kubernetes.io/projected/032ee506-19b6-43c8-9d49-f1cf24f19f4d-kube-api-access-5qvxw\") pod \"authorino-f55b4dd8-2qz9c\" (UID: \"032ee506-19b6-43c8-9d49-f1cf24f19f4d\") " pod="kuadrant-system/authorino-f55b4dd8-2qz9c" Jul 10 17:53:28.297794 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.297730 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/032ee506-19b6-43c8-9d49-f1cf24f19f4d-tls-cert\") pod \"authorino-f55b4dd8-2qz9c\" (UID: \"032ee506-19b6-43c8-9d49-f1cf24f19f4d\") " pod="kuadrant-system/authorino-f55b4dd8-2qz9c" Jul 10 17:53:28.398412 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.398372 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvxw\" (UniqueName: \"kubernetes.io/projected/032ee506-19b6-43c8-9d49-f1cf24f19f4d-kube-api-access-5qvxw\") pod \"authorino-f55b4dd8-2qz9c\" (UID: \"032ee506-19b6-43c8-9d49-f1cf24f19f4d\") " pod="kuadrant-system/authorino-f55b4dd8-2qz9c" Jul 10 17:53:28.398611 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.398428 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/032ee506-19b6-43c8-9d49-f1cf24f19f4d-tls-cert\") pod \"authorino-f55b4dd8-2qz9c\" (UID: \"032ee506-19b6-43c8-9d49-f1cf24f19f4d\") " pod="kuadrant-system/authorino-f55b4dd8-2qz9c" Jul 10 17:53:28.400999 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.400979 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/032ee506-19b6-43c8-9d49-f1cf24f19f4d-tls-cert\") pod \"authorino-f55b4dd8-2qz9c\" (UID: \"032ee506-19b6-43c8-9d49-f1cf24f19f4d\") " pod="kuadrant-system/authorino-f55b4dd8-2qz9c" Jul 10 17:53:28.406062 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.406039 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvxw\" (UniqueName: \"kubernetes.io/projected/032ee506-19b6-43c8-9d49-f1cf24f19f4d-kube-api-access-5qvxw\") pod \"authorino-f55b4dd8-2qz9c\" (UID: \"032ee506-19b6-43c8-9d49-f1cf24f19f4d\") " pod="kuadrant-system/authorino-f55b4dd8-2qz9c" Jul 10 17:53:28.507441 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.507341 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f55b4dd8-2qz9c" Jul 10 17:53:28.647962 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.647937 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f55b4dd8-2qz9c"] Jul 10 17:53:28.650556 ip-10-0-137-103 kubenswrapper[2556]: W0710 17:53:28.650532 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod032ee506_19b6_43c8_9d49_f1cf24f19f4d.slice/crio-1eb0b3ae6822b5b50212edaae76dd58f1bfa72b10d75669c0fbbd254d3b24220 WatchSource:0}: Error finding container 1eb0b3ae6822b5b50212edaae76dd58f1bfa72b10d75669c0fbbd254d3b24220: Status 404 returned error can't find the container with id 1eb0b3ae6822b5b50212edaae76dd58f1bfa72b10d75669c0fbbd254d3b24220 Jul 10 17:53:28.963603 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:28.963570 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f55b4dd8-2qz9c" event={"ID":"032ee506-19b6-43c8-9d49-f1cf24f19f4d","Type":"ContainerStarted","Data":"1eb0b3ae6822b5b50212edaae76dd58f1bfa72b10d75669c0fbbd254d3b24220"} Jul 10 17:53:29.970134 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:29.970100 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f55b4dd8-2qz9c" event={"ID":"032ee506-19b6-43c8-9d49-f1cf24f19f4d","Type":"ContainerStarted","Data":"9db1fa9ee983bec6d93d0b3c453a50304ee6cfeedbddbe37c9026ab7ccefd702"} Jul 10 17:53:29.985896 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:29.985845 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f55b4dd8-2qz9c" podStartSLOduration=1.621410366 podStartE2EDuration="1.985827092s" totalImagesPullingTime="364.416726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:53:29.984122841 +0000 UTC m=+859.258449532" watchObservedRunningTime="2026-07-10 17:53:29.985827092 +0000 UTC m=+859.260153786" Jul 10 17:53:57.350728 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:53:57.350685 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:54:06.960178 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:54:06.960087 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:54:11.154594 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:54:11.154567 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 17:54:11.155112 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:54:11.154701 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 17:54:17.854745 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:54:17.854706 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:54:26.857066 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:54:26.857029 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:54:37.747692 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:54:37.747659 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:55:04.157131 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:55:04.157081 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:55:16.259631 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:55:16.259592 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:55:54.673149 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:55:54.673066 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:56:04.458219 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:56:04.458179 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:56:13.570183 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:56:13.570149 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:56:24.771979 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:56:24.771927 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:56:36.565854 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:56:36.565812 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:56:46.004710 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:56:46.004667 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:56:57.470154 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:56:57.470113 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:57:07.221909 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:57:07.221828 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:57:29.675108 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:57:29.675069 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:57:38.578099 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:57:38.578064 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:58:02.558991 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:58:02.558933 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:58:13.650547 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:58:13.650505 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:58:29.864547 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:58:29.864501 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:58:39.256546 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:58:39.256467 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:58:48.662182 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:58:48.662142 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:58:58.252861 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:58:58.252810 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:59:07.160593 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:59:07.160555 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 17:59:11.191007 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:59:11.190959 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 17:59:11.192634 ip-10-0-137-103 kubenswrapper[2556]: I0710 17:59:11.192610 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 18:00:56.159999 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:00:56.159938 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:01:07.556216 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:01:07.556174 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:01:16.058224 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:01:16.058191 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:01:27.856863 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:01:27.856823 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:01:39.749353 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:01:39.749262 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:01:50.059208 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:01:50.059163 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:01:59.370931 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:01:59.370878 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:02:09.804660 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:02:09.804622 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:02:20.364528 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:02:20.364490 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:02:30.755354 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:02:30.755320 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:02:44.153859 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:02:44.153828 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:03:02.365851 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:03:02.365757 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:03:11.368345 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:03:11.368312 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:03:22.862959 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:03:22.862912 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:03:35.557882 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:03:35.557845 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:03:51.664615 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:03:51.664581 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:04:02.882318 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:04:02.882279 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:04:11.220287 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:04:11.220248 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 18:04:11.222231 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:04:11.222204 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 18:04:12.155347 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:04:12.155300 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:04:22.366832 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:04:22.366796 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:04:31.964185 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:04:31.964103 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:04:41.863112 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:04:41.863078 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:04:55.353354 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:04:55.353320 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:05:04.562776 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:05:04.562736 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:05:19.055127 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:05:19.055089 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:05:27.161269 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:05:27.161225 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:05:39.364021 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:05:39.363987 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:05:55.464652 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:05:55.464615 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:06:07.458655 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:06:07.458543 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:06:18.850254 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:06:18.850211 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:06:27.601242 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:06:27.601205 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:06:48.765042 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:06:48.765005 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:06:58.982390 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:06:58.982348 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:07:11.973320 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:07:11.973281 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:07:23.877251 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:07:23.877215 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:09:03.578091 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:09:03.577995 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:09:11.249202 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:09:11.249173 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 18:09:11.251504 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:09:11.251479 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 18:09:12.564841 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:09:12.564805 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:09:27.968433 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:09:27.968393 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:09:39.279895 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:09:39.279856 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:10:09.778741 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:09.778706 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:10:12.686585 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.686548 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb"] Jul 10 18:10:12.689875 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.689848 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.692776 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.692752 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ba6e3fc9\"/\"kube-root-ca.crt\"" Jul 10 18:10:12.692918 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.692827 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ba6e3fc9\"/\"default-dockercfg-5gqtm\"" Jul 10 18:10:12.693840 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.693816 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ba6e3fc9\"/\"openshift-service-ca.crt\"" Jul 10 18:10:12.693946 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.693873 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ba6e3fc9\"/\"e2e-lifecycle-model-ba6e3fc9-kserve-self-signed-certs\"" Jul 10 18:10:12.700957 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.700930 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb"] Jul 10 18:10:12.808603 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.808561 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-tls-certs\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.808806 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.808631 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-home\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.808806 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.808679 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnbt8\" (UniqueName: \"kubernetes.io/projected/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-kube-api-access-tnbt8\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.808806 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.808746 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-tmp-dir\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.808806 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.808800 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-model-cache\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.809029 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.808837 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-dshm\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.910052 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.909995 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbt8\" (UniqueName: \"kubernetes.io/projected/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-kube-api-access-tnbt8\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.910254 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.910087 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-tmp-dir\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.910254 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.910126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-model-cache\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.910254 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.910146 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-dshm\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.910254 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.910185 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-tls-certs\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.910254 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.910212 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-home\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.910516 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.910497 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-tmp-dir\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.910568 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.910548 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-model-cache\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.910646 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.910610 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-home\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.912589 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.912572 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-dshm\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.912847 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.912829 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-tls-certs\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:12.918413 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:12.918392 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbt8\" (UniqueName: \"kubernetes.io/projected/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-kube-api-access-tnbt8\") pod \"e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:13.000258 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:13.000154 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:13.155724 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:13.155694 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb"] Jul 10 18:10:13.158171 ip-10-0-137-103 kubenswrapper[2556]: W0710 18:10:13.158141 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod980e5532_7c02_48e6_a608_3bd8e0f5a1f9.slice/crio-523c7542703687d31bb7f6c2cdcbebb9d41cce4082fc2920fcd47f6673fe5438 WatchSource:0}: Error finding container 523c7542703687d31bb7f6c2cdcbebb9d41cce4082fc2920fcd47f6673fe5438: Status 404 returned error can't find the container with id 523c7542703687d31bb7f6c2cdcbebb9d41cce4082fc2920fcd47f6673fe5438 Jul 10 18:10:13.937122 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:13.937079 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" event={"ID":"980e5532-7c02-48e6-a608-3bd8e0f5a1f9","Type":"ContainerStarted","Data":"cb533483142a8fec3f5f80ffcf9576397729ca46a580cd303c7dc4e41674f842"} Jul 10 18:10:13.937122 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:13.937122 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" event={"ID":"980e5532-7c02-48e6-a608-3bd8e0f5a1f9","Type":"ContainerStarted","Data":"523c7542703687d31bb7f6c2cdcbebb9d41cce4082fc2920fcd47f6673fe5438"} Jul 10 18:10:13.963005 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:13.962934 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" podStartSLOduration=1.962917564 podStartE2EDuration="1.962917564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:10:13.961018331 +0000 UTC m=+1863.235345056" watchObservedRunningTime="2026-07-10 18:10:13.962917564 +0000 UTC m=+1863.237244259" Jul 10 18:10:23.000891 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:23.000840 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:23.001366 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:23.000913 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:23.014113 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:23.014078 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:23.995588 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:23.995558 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:10:25.274906 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:25.274869 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:10:37.973404 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:37.973287 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:10:51.836894 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:51.836857 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb"] Jul 10 18:10:52.567502 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:10:52.567465 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:11:06.858271 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:06.858202 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" podUID="980e5532-7c02-48e6-a608-3bd8e0f5a1f9" containerName="main" containerID="cri-o://cb533483142a8fec3f5f80ffcf9576397729ca46a580cd303c7dc4e41674f842" gracePeriod=45 Jul 10 18:11:07.118764 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.118700 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:11:07.123562 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.123536 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-home\") pod \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " Jul 10 18:11:07.123685 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.123611 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-tmp-dir\") pod \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " Jul 10 18:11:07.123685 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.123657 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnbt8\" (UniqueName: \"kubernetes.io/projected/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-kube-api-access-tnbt8\") pod \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " Jul 10 18:11:07.123804 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.123703 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-model-cache\") pod \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " Jul 10 18:11:07.123804 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.123740 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-dshm\") pod \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " Jul 10 18:11:07.123904 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.123793 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-home" (OuterVolumeSpecName: "home") pod "980e5532-7c02-48e6-a608-3bd8e0f5a1f9" (UID: "980e5532-7c02-48e6-a608-3bd8e0f5a1f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:11:07.123904 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.123823 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "980e5532-7c02-48e6-a608-3bd8e0f5a1f9" (UID: "980e5532-7c02-48e6-a608-3bd8e0f5a1f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:11:07.123904 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.123827 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-tls-certs\") pod \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\" (UID: \"980e5532-7c02-48e6-a608-3bd8e0f5a1f9\") " Jul 10 18:11:07.124086 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.123956 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "980e5532-7c02-48e6-a608-3bd8e0f5a1f9" (UID: "980e5532-7c02-48e6-a608-3bd8e0f5a1f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:11:07.124142 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.124114 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-home\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:11:07.124142 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.124128 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-tmp-dir\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:11:07.124142 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.124138 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-model-cache\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:11:07.126016 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.125998 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "980e5532-7c02-48e6-a608-3bd8e0f5a1f9" (UID: "980e5532-7c02-48e6-a608-3bd8e0f5a1f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:11:07.126016 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.126004 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-dshm" (OuterVolumeSpecName: "dshm") pod "980e5532-7c02-48e6-a608-3bd8e0f5a1f9" (UID: "980e5532-7c02-48e6-a608-3bd8e0f5a1f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:11:07.126128 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.126108 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-kube-api-access-tnbt8" (OuterVolumeSpecName: "kube-api-access-tnbt8") pod "980e5532-7c02-48e6-a608-3bd8e0f5a1f9" (UID: "980e5532-7c02-48e6-a608-3bd8e0f5a1f9"). InnerVolumeSpecName "kube-api-access-tnbt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:11:07.200091 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.200059 2556 generic.go:356] "Generic (PLEG): container finished" podID="980e5532-7c02-48e6-a608-3bd8e0f5a1f9" containerID="cb533483142a8fec3f5f80ffcf9576397729ca46a580cd303c7dc4e41674f842" exitCode=0 Jul 10 18:11:07.200235 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.200103 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" Jul 10 18:11:07.200235 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.200139 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" event={"ID":"980e5532-7c02-48e6-a608-3bd8e0f5a1f9","Type":"ContainerDied","Data":"cb533483142a8fec3f5f80ffcf9576397729ca46a580cd303c7dc4e41674f842"} Jul 10 18:11:07.200235 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.200190 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb" event={"ID":"980e5532-7c02-48e6-a608-3bd8e0f5a1f9","Type":"ContainerDied","Data":"523c7542703687d31bb7f6c2cdcbebb9d41cce4082fc2920fcd47f6673fe5438"} Jul 10 18:11:07.200235 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.200211 2556 scope.go:117] "RemoveContainer" containerID="cb533483142a8fec3f5f80ffcf9576397729ca46a580cd303c7dc4e41674f842" Jul 10 18:11:07.210421 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.210399 2556 scope.go:117] "RemoveContainer" containerID="cb533483142a8fec3f5f80ffcf9576397729ca46a580cd303c7dc4e41674f842" Jul 10 18:11:07.210735 ip-10-0-137-103 kubenswrapper[2556]: E0710 18:11:07.210712 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb533483142a8fec3f5f80ffcf9576397729ca46a580cd303c7dc4e41674f842\": container with ID starting with cb533483142a8fec3f5f80ffcf9576397729ca46a580cd303c7dc4e41674f842 not found: ID does not exist" containerID="cb533483142a8fec3f5f80ffcf9576397729ca46a580cd303c7dc4e41674f842" Jul 10 18:11:07.210839 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.210740 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb533483142a8fec3f5f80ffcf9576397729ca46a580cd303c7dc4e41674f842"} err="failed to get container status \"cb533483142a8fec3f5f80ffcf9576397729ca46a580cd303c7dc4e41674f842\": rpc error: code = NotFound desc = could not find container \"cb533483142a8fec3f5f80ffcf9576397729ca46a580cd303c7dc4e41674f842\": container with ID starting with cb533483142a8fec3f5f80ffcf9576397729ca46a580cd303c7dc4e41674f842 not found: ID does not exist" Jul 10 18:11:07.222748 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.222723 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb"] Jul 10 18:11:07.225261 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.225245 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-tls-certs\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:11:07.225314 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.225264 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnbt8\" (UniqueName: \"kubernetes.io/projected/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-kube-api-access-tnbt8\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:11:07.225314 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.225273 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/980e5532-7c02-48e6-a608-3bd8e0f5a1f9-dshm\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:11:07.228624 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.228598 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ba6e3fc9/e2e-lifecycle-model-ba6e3fc9-kserve-6d59d96799-xzzmb"] Jul 10 18:11:07.254988 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:07.254935 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="980e5532-7c02-48e6-a608-3bd8e0f5a1f9" path="/var/lib/kubelet/pods/980e5532-7c02-48e6-a608-3bd8e0f5a1f9/volumes" Jul 10 18:11:31.623325 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:11:31.623276 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:12:42.270082 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:12:42.270040 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:12:56.858240 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:12:56.858199 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:13:08.768084 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.768041 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q"] Jul 10 18:13:08.768720 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.768591 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="980e5532-7c02-48e6-a608-3bd8e0f5a1f9" containerName="main" Jul 10 18:13:08.768720 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.768613 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="980e5532-7c02-48e6-a608-3bd8e0f5a1f9" containerName="main" Jul 10 18:13:08.768720 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.768712 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="980e5532-7c02-48e6-a608-3bd8e0f5a1f9" containerName="main" Jul 10 18:13:08.771709 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.771687 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.776364 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.776344 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b1e501\"/\"kube-root-ca.crt\"" Jul 10 18:13:08.777512 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.777492 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b1e501\"/\"test-model-b1e501-kserve-self-signed-certs\"" Jul 10 18:13:08.777639 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.777518 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b1e501\"/\"default-dockercfg-m7br9\"" Jul 10 18:13:08.777639 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.777585 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b1e501\"/\"openshift-service-ca.crt\"" Jul 10 18:13:08.783139 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.783113 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q"] Jul 10 18:13:08.819306 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.819271 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-model-cache\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.819498 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.819328 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-tls-certs\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.819498 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.819433 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-home\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.819498 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.819471 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz59p\" (UniqueName: \"kubernetes.io/projected/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-kube-api-access-vz59p\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.819624 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.819509 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-dshm\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.819624 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.819577 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-tmp-dir\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.920947 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.920908 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-dshm\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.921162 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.921000 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-tmp-dir\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.921162 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.921101 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-model-cache\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.921162 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.921140 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-tls-certs\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.921337 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.921182 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-home\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.921337 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.921200 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz59p\" (UniqueName: \"kubernetes.io/projected/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-kube-api-access-vz59p\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.921490 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.921469 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-tmp-dir\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.921534 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.921489 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-model-cache\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.921576 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.921545 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-home\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.923353 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.923330 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-dshm\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.923753 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.923737 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-tls-certs\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:08.929093 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:08.929068 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz59p\" (UniqueName: \"kubernetes.io/projected/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-kube-api-access-vz59p\") pod \"test-model-b1e501-kserve-7457dcf765-ndw5q\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:09.082160 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:09.082066 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:09.224402 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:09.224376 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q"] Jul 10 18:13:09.227030 ip-10-0-137-103 kubenswrapper[2556]: W0710 18:13:09.226997 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbab7260f_0e6b_4ca8_8a5c_a3b91f7bef45.slice/crio-7d0866f9e0691215881608b9515135ad53899c140fd751cdc476554ec08f7b3a WatchSource:0}: Error finding container 7d0866f9e0691215881608b9515135ad53899c140fd751cdc476554ec08f7b3a: Status 404 returned error can't find the container with id 7d0866f9e0691215881608b9515135ad53899c140fd751cdc476554ec08f7b3a Jul 10 18:13:09.791247 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:09.791207 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" event={"ID":"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45","Type":"ContainerStarted","Data":"a48e53dbc46374b445fcc14b03a0bdbc750b793f5ff4b9405551363b412e871d"} Jul 10 18:13:09.791247 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:09.791250 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" event={"ID":"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45","Type":"ContainerStarted","Data":"7d0866f9e0691215881608b9515135ad53899c140fd751cdc476554ec08f7b3a"} Jul 10 18:13:09.810373 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:09.810320 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" podStartSLOduration=1.810301693 podStartE2EDuration="1.810301693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:13:09.809427607 +0000 UTC m=+2039.083754302" watchObservedRunningTime="2026-07-10 18:13:09.810301693 +0000 UTC m=+2039.084628387" Jul 10 18:13:12.168357 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:12.168316 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:13:19.083007 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:19.082941 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:19.083486 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:19.083034 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:19.095481 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:19.095453 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:19.850751 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:19.850717 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:13:26.060518 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:26.060480 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:13:35.561640 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.561566 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz"] Jul 10 18:13:35.565372 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.565352 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.567998 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.567959 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3fac88\"/\"kube-root-ca.crt\"" Jul 10 18:13:35.568995 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.568946 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3fac88\"/\"openshift-service-ca.crt\"" Jul 10 18:13:35.569076 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.569056 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3fac88\"/\"test-model-3fac88-kserve-self-signed-certs\"" Jul 10 18:13:35.569133 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.569056 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3fac88\"/\"default-dockercfg-nxrq2\"" Jul 10 18:13:35.575609 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.575580 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz"] Jul 10 18:13:35.667159 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.667119 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5274f050-30b9-4a57-8efa-f853995454df-tls-certs\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.667159 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.667168 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-dshm\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.667373 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.667190 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-model-cache\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.667373 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.667258 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-tmp-dir\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.667373 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.667299 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-home\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.667373 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.667342 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj6f7\" (UniqueName: \"kubernetes.io/projected/5274f050-30b9-4a57-8efa-f853995454df-kube-api-access-kj6f7\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.768545 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.768506 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-model-cache\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.768545 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.768551 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-tmp-dir\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.768765 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.768573 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-home\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.768765 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.768601 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6f7\" (UniqueName: \"kubernetes.io/projected/5274f050-30b9-4a57-8efa-f853995454df-kube-api-access-kj6f7\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.768765 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.768654 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5274f050-30b9-4a57-8efa-f853995454df-tls-certs\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.768765 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.768682 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-dshm\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.768997 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.768956 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-model-cache\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.769053 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.769029 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-home\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.769099 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.769050 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-tmp-dir\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.771201 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.771169 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-dshm\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.771426 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.771404 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5274f050-30b9-4a57-8efa-f853995454df-tls-certs\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.779907 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.779880 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj6f7\" (UniqueName: \"kubernetes.io/projected/5274f050-30b9-4a57-8efa-f853995454df-kube-api-access-kj6f7\") pod \"test-model-3fac88-kserve-fc9459995-gzgmz\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:35.875138 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:35.875107 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:36.027462 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:36.027430 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz"] Jul 10 18:13:36.029677 ip-10-0-137-103 kubenswrapper[2556]: W0710 18:13:36.029647 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5274f050_30b9_4a57_8efa_f853995454df.slice/crio-986022adc1a10cc309095a94e6a3288ff6ea1a3a018a47582a3c2ef48988870d WatchSource:0}: Error finding container 986022adc1a10cc309095a94e6a3288ff6ea1a3a018a47582a3c2ef48988870d: Status 404 returned error can't find the container with id 986022adc1a10cc309095a94e6a3288ff6ea1a3a018a47582a3c2ef48988870d Jul 10 18:13:36.922159 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:36.922123 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" event={"ID":"5274f050-30b9-4a57-8efa-f853995454df","Type":"ContainerStarted","Data":"ba78331c4fd5d25f21f103e7edcbb1750a230d3e70f06257be8fdd5f23119f3b"} Jul 10 18:13:36.922159 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:36.922166 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" event={"ID":"5274f050-30b9-4a57-8efa-f853995454df","Type":"ContainerStarted","Data":"986022adc1a10cc309095a94e6a3288ff6ea1a3a018a47582a3c2ef48988870d"} Jul 10 18:13:36.942801 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:36.942744 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" podStartSLOduration=1.942722802 podStartE2EDuration="1.942722802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:13:36.942526361 +0000 UTC m=+2066.216853060" watchObservedRunningTime="2026-07-10 18:13:36.942722802 +0000 UTC m=+2066.217049496" Jul 10 18:13:40.568195 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:40.568163 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:13:45.875462 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:45.875416 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:45.875920 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:45.875486 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:45.888250 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:45.888218 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:45.975591 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:45.975559 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:13:55.470945 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:13:55.470905 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:14:01.146530 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:01.146493 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q"] Jul 10 18:14:02.087411 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:02.087370 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz"] Jul 10 18:14:08.474294 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:08.474259 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:14:11.283959 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:11.283926 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 18:14:11.286667 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:11.286628 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 18:14:16.168221 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.168167 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" podUID="bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45" containerName="main" containerID="cri-o://a48e53dbc46374b445fcc14b03a0bdbc750b793f5ff4b9405551363b412e871d" gracePeriod=45 Jul 10 18:14:16.419844 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.419782 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:14:16.465812 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.465774 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-home\") pod \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " Jul 10 18:14:16.466039 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.465846 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-tmp-dir\") pod \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " Jul 10 18:14:16.466039 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.465868 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-dshm\") pod \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " Jul 10 18:14:16.466039 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.465922 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-model-cache\") pod \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " Jul 10 18:14:16.466039 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.466015 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-tls-certs\") pod \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " Jul 10 18:14:16.466281 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.466071 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz59p\" (UniqueName: \"kubernetes.io/projected/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-kube-api-access-vz59p\") pod \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\" (UID: \"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45\") " Jul 10 18:14:16.466281 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.466113 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-home" (OuterVolumeSpecName: "home") pod "bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45" (UID: "bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:16.466281 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.466124 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45" (UID: "bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:16.466281 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.466184 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-model-cache" (OuterVolumeSpecName: "model-cache") pod "bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45" (UID: "bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:16.466496 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.466348 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-model-cache\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:14:16.466496 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.466361 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-home\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:14:16.466496 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.466370 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-tmp-dir\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:14:16.468434 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.468403 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-dshm" (OuterVolumeSpecName: "dshm") pod "bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45" (UID: "bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:16.468434 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.468420 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45" (UID: "bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:14:16.468598 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.468408 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-kube-api-access-vz59p" (OuterVolumeSpecName: "kube-api-access-vz59p") pod "bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45" (UID: "bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45"). InnerVolumeSpecName "kube-api-access-vz59p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:14:16.567804 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.567768 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-tls-certs\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:14:16.567804 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.567798 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz59p\" (UniqueName: \"kubernetes.io/projected/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-kube-api-access-vz59p\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:14:16.567804 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:16.567809 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45-dshm\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:14:17.108778 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.108734 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" podUID="5274f050-30b9-4a57-8efa-f853995454df" containerName="main" containerID="cri-o://ba78331c4fd5d25f21f103e7edcbb1750a230d3e70f06257be8fdd5f23119f3b" gracePeriod=45 Jul 10 18:14:17.120081 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.120051 2556 generic.go:356] "Generic (PLEG): container finished" podID="bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45" containerID="a48e53dbc46374b445fcc14b03a0bdbc750b793f5ff4b9405551363b412e871d" exitCode=0 Jul 10 18:14:17.120203 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.120099 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" Jul 10 18:14:17.120203 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.120130 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" event={"ID":"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45","Type":"ContainerDied","Data":"a48e53dbc46374b445fcc14b03a0bdbc750b793f5ff4b9405551363b412e871d"} Jul 10 18:14:17.120203 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.120179 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q" event={"ID":"bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45","Type":"ContainerDied","Data":"7d0866f9e0691215881608b9515135ad53899c140fd751cdc476554ec08f7b3a"} Jul 10 18:14:17.120203 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.120201 2556 scope.go:117] "RemoveContainer" containerID="a48e53dbc46374b445fcc14b03a0bdbc750b793f5ff4b9405551363b412e871d" Jul 10 18:14:17.183199 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.182860 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q"] Jul 10 18:14:17.186186 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.186161 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b1e501/test-model-b1e501-kserve-7457dcf765-ndw5q"] Jul 10 18:14:17.220548 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.220420 2556 scope.go:117] "RemoveContainer" containerID="a48e53dbc46374b445fcc14b03a0bdbc750b793f5ff4b9405551363b412e871d" Jul 10 18:14:17.220807 ip-10-0-137-103 kubenswrapper[2556]: E0710 18:14:17.220787 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48e53dbc46374b445fcc14b03a0bdbc750b793f5ff4b9405551363b412e871d\": container with ID starting with a48e53dbc46374b445fcc14b03a0bdbc750b793f5ff4b9405551363b412e871d not found: ID does not exist" containerID="a48e53dbc46374b445fcc14b03a0bdbc750b793f5ff4b9405551363b412e871d" Jul 10 18:14:17.220873 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.220817 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48e53dbc46374b445fcc14b03a0bdbc750b793f5ff4b9405551363b412e871d"} err="failed to get container status \"a48e53dbc46374b445fcc14b03a0bdbc750b793f5ff4b9405551363b412e871d\": rpc error: code = NotFound desc = could not find container \"a48e53dbc46374b445fcc14b03a0bdbc750b793f5ff4b9405551363b412e871d\": container with ID starting with a48e53dbc46374b445fcc14b03a0bdbc750b793f5ff4b9405551363b412e871d not found: ID does not exist" Jul 10 18:14:17.255571 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.255538 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45" path="/var/lib/kubelet/pods/bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45/volumes" Jul 10 18:14:17.373842 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.373771 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:14:17.477057 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.477020 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-model-cache\") pod \"5274f050-30b9-4a57-8efa-f853995454df\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " Jul 10 18:14:17.477243 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.477082 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj6f7\" (UniqueName: \"kubernetes.io/projected/5274f050-30b9-4a57-8efa-f853995454df-kube-api-access-kj6f7\") pod \"5274f050-30b9-4a57-8efa-f853995454df\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " Jul 10 18:14:17.477243 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.477123 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-dshm\") pod \"5274f050-30b9-4a57-8efa-f853995454df\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " Jul 10 18:14:17.477243 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.477152 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5274f050-30b9-4a57-8efa-f853995454df-tls-certs\") pod \"5274f050-30b9-4a57-8efa-f853995454df\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " Jul 10 18:14:17.477243 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.477184 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-home\") pod \"5274f050-30b9-4a57-8efa-f853995454df\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " Jul 10 18:14:17.477243 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.477219 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-tmp-dir\") pod \"5274f050-30b9-4a57-8efa-f853995454df\" (UID: \"5274f050-30b9-4a57-8efa-f853995454df\") " Jul 10 18:14:17.477491 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.477372 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-model-cache" (OuterVolumeSpecName: "model-cache") pod "5274f050-30b9-4a57-8efa-f853995454df" (UID: "5274f050-30b9-4a57-8efa-f853995454df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:17.477595 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.477527 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-home" (OuterVolumeSpecName: "home") pod "5274f050-30b9-4a57-8efa-f853995454df" (UID: "5274f050-30b9-4a57-8efa-f853995454df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:17.477595 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.477550 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-model-cache\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:14:17.477595 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.477562 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5274f050-30b9-4a57-8efa-f853995454df" (UID: "5274f050-30b9-4a57-8efa-f853995454df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:17.479685 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.479655 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5274f050-30b9-4a57-8efa-f853995454df-kube-api-access-kj6f7" (OuterVolumeSpecName: "kube-api-access-kj6f7") pod "5274f050-30b9-4a57-8efa-f853995454df" (UID: "5274f050-30b9-4a57-8efa-f853995454df"). InnerVolumeSpecName "kube-api-access-kj6f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:14:17.479685 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.479681 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5274f050-30b9-4a57-8efa-f853995454df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5274f050-30b9-4a57-8efa-f853995454df" (UID: "5274f050-30b9-4a57-8efa-f853995454df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:14:17.479829 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.479696 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-dshm" (OuterVolumeSpecName: "dshm") pod "5274f050-30b9-4a57-8efa-f853995454df" (UID: "5274f050-30b9-4a57-8efa-f853995454df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:17.578547 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.578505 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj6f7\" (UniqueName: \"kubernetes.io/projected/5274f050-30b9-4a57-8efa-f853995454df-kube-api-access-kj6f7\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:14:17.578547 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.578538 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-dshm\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:14:17.578547 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.578549 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5274f050-30b9-4a57-8efa-f853995454df-tls-certs\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:14:17.578547 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.578560 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-home\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:14:17.578816 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:17.578568 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5274f050-30b9-4a57-8efa-f853995454df-tmp-dir\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:14:18.126470 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:18.126432 2556 generic.go:356] "Generic (PLEG): container finished" podID="5274f050-30b9-4a57-8efa-f853995454df" containerID="ba78331c4fd5d25f21f103e7edcbb1750a230d3e70f06257be8fdd5f23119f3b" exitCode=0 Jul 10 18:14:18.126640 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:18.126526 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" Jul 10 18:14:18.126640 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:18.126577 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" event={"ID":"5274f050-30b9-4a57-8efa-f853995454df","Type":"ContainerDied","Data":"ba78331c4fd5d25f21f103e7edcbb1750a230d3e70f06257be8fdd5f23119f3b"} Jul 10 18:14:18.126640 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:18.126615 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz" event={"ID":"5274f050-30b9-4a57-8efa-f853995454df","Type":"ContainerDied","Data":"986022adc1a10cc309095a94e6a3288ff6ea1a3a018a47582a3c2ef48988870d"} Jul 10 18:14:18.126640 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:18.126633 2556 scope.go:117] "RemoveContainer" containerID="ba78331c4fd5d25f21f103e7edcbb1750a230d3e70f06257be8fdd5f23119f3b" Jul 10 18:14:18.136112 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:18.136099 2556 scope.go:117] "RemoveContainer" containerID="ba78331c4fd5d25f21f103e7edcbb1750a230d3e70f06257be8fdd5f23119f3b" Jul 10 18:14:18.136375 ip-10-0-137-103 kubenswrapper[2556]: E0710 18:14:18.136352 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba78331c4fd5d25f21f103e7edcbb1750a230d3e70f06257be8fdd5f23119f3b\": container with ID starting with ba78331c4fd5d25f21f103e7edcbb1750a230d3e70f06257be8fdd5f23119f3b not found: ID does not exist" containerID="ba78331c4fd5d25f21f103e7edcbb1750a230d3e70f06257be8fdd5f23119f3b" Jul 10 18:14:18.136449 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:18.136381 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba78331c4fd5d25f21f103e7edcbb1750a230d3e70f06257be8fdd5f23119f3b"} err="failed to get container status \"ba78331c4fd5d25f21f103e7edcbb1750a230d3e70f06257be8fdd5f23119f3b\": rpc error: code = NotFound desc = could not find container \"ba78331c4fd5d25f21f103e7edcbb1750a230d3e70f06257be8fdd5f23119f3b\": container with ID starting with ba78331c4fd5d25f21f103e7edcbb1750a230d3e70f06257be8fdd5f23119f3b not found: ID does not exist" Jul 10 18:14:18.149150 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:18.149111 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz"] Jul 10 18:14:18.153340 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:18.153305 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3fac88/test-model-3fac88-kserve-fc9459995-gzgmz"] Jul 10 18:14:18.175063 ip-10-0-137-103 kubenswrapper[2556]: E0710 18:14:18.175022 2556 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5274f050_30b9_4a57_8efa_f853995454df.slice/crio-986022adc1a10cc309095a94e6a3288ff6ea1a3a018a47582a3c2ef48988870d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5274f050_30b9_4a57_8efa_f853995454df.slice\": RecentStats: unable to find data in memory cache]" Jul 10 18:14:19.255072 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:19.255034 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5274f050-30b9-4a57-8efa-f853995454df" path="/var/lib/kubelet/pods/5274f050-30b9-4a57-8efa-f853995454df/volumes" Jul 10 18:14:49.662030 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:14:49.661987 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:15:05.410446 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:15:05.410362 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:15:21.166873 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:15:21.166837 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:15:34.671409 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:15:34.671356 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:15:50.564620 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:15:50.564584 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:16:07.172172 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.172132 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh"] Jul 10 18:16:07.172683 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.172663 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5274f050-30b9-4a57-8efa-f853995454df" containerName="main" Jul 10 18:16:07.172766 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.172687 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5274f050-30b9-4a57-8efa-f853995454df" containerName="main" Jul 10 18:16:07.172766 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.172702 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45" containerName="main" Jul 10 18:16:07.172766 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.172711 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45" containerName="main" Jul 10 18:16:07.172910 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.172824 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5274f050-30b9-4a57-8efa-f853995454df" containerName="main" Jul 10 18:16:07.172910 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.172840 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bab7260f-0e6b-4ca8-8a5c-a3b91f7bef45" containerName="main" Jul 10 18:16:07.175927 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.175904 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.180337 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.180313 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b8bbaf\"/\"kube-root-ca.crt\"" Jul 10 18:16:07.180473 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.180342 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b8bbaf\"/\"auth-test-model-b8bbaf-kserve-self-signed-certs\"" Jul 10 18:16:07.180473 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.180460 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b8bbaf\"/\"default-dockercfg-sdccd\"" Jul 10 18:16:07.180569 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.180552 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b8bbaf\"/\"openshift-service-ca.crt\"" Jul 10 18:16:07.191174 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.191148 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh"] Jul 10 18:16:07.229843 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.229807 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjbt2\" (UniqueName: \"kubernetes.io/projected/edcef346-3dd0-4941-bc54-ae54857706e8-kube-api-access-gjbt2\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.230050 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.229850 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-home\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.230050 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.229896 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-dshm\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.230050 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.229912 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-tmp-dir\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.230050 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.229926 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edcef346-3dd0-4941-bc54-ae54857706e8-tls-certs\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.230050 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.230001 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-model-cache\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.330467 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.330427 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-model-cache\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.330650 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.330551 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjbt2\" (UniqueName: \"kubernetes.io/projected/edcef346-3dd0-4941-bc54-ae54857706e8-kube-api-access-gjbt2\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.330650 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.330592 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-home\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.330650 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.330629 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-dshm\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.330804 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.330654 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-tmp-dir\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.330804 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.330677 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edcef346-3dd0-4941-bc54-ae54857706e8-tls-certs\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.330929 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.330904 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-model-cache\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.331026 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.331008 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-home\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.331095 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.331045 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-tmp-dir\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.333160 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.333134 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-dshm\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.333418 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.333399 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edcef346-3dd0-4941-bc54-ae54857706e8-tls-certs\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.339466 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.339433 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjbt2\" (UniqueName: \"kubernetes.io/projected/edcef346-3dd0-4941-bc54-ae54857706e8-kube-api-access-gjbt2\") pod \"auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.485204 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.485106 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:07.626238 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.626210 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh"] Jul 10 18:16:07.628845 ip-10-0-137-103 kubenswrapper[2556]: W0710 18:16:07.628818 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedcef346_3dd0_4941_bc54_ae54857706e8.slice/crio-01e7885ac8220ebe1d9c45a4c3b1fc82a35e1f7558b2cdc9bc5269bb08cdc5e8 WatchSource:0}: Error finding container 01e7885ac8220ebe1d9c45a4c3b1fc82a35e1f7558b2cdc9bc5269bb08cdc5e8: Status 404 returned error can't find the container with id 01e7885ac8220ebe1d9c45a4c3b1fc82a35e1f7558b2cdc9bc5269bb08cdc5e8 Jul 10 18:16:07.651919 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:07.651888 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" event={"ID":"edcef346-3dd0-4941-bc54-ae54857706e8","Type":"ContainerStarted","Data":"01e7885ac8220ebe1d9c45a4c3b1fc82a35e1f7558b2cdc9bc5269bb08cdc5e8"} Jul 10 18:16:08.658375 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:08.658340 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" event={"ID":"edcef346-3dd0-4941-bc54-ae54857706e8","Type":"ContainerStarted","Data":"3e3f6c8abbbfc285060ef68cdf97ee2fc456b3d486ef858f69f3ddeae1599fee"} Jul 10 18:16:08.682709 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:08.682663 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" podStartSLOduration=1.68264557 podStartE2EDuration="1.68264557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:16:08.680816185 +0000 UTC m=+2217.955142879" watchObservedRunningTime="2026-07-10 18:16:08.68264557 +0000 UTC m=+2217.956972264" Jul 10 18:16:10.582940 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:10.582905 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:16:17.486036 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:17.485985 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:17.486419 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:17.486059 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:17.498782 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:17.498753 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:17.715735 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:17.715700 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:16:36.062124 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.062018 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds"] Jul 10 18:16:36.065898 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.065876 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.069594 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.069570 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0af48d\"/\"default-dockercfg-jdhgv\"" Jul 10 18:16:36.069721 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.069577 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0af48d\"/\"auth-test-model-0af48d-kserve-self-signed-certs\"" Jul 10 18:16:36.069721 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.069605 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0af48d\"/\"openshift-service-ca.crt\"" Jul 10 18:16:36.069721 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.069651 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0af48d\"/\"kube-root-ca.crt\"" Jul 10 18:16:36.075796 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.075774 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds"] Jul 10 18:16:36.193030 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.192961 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-home\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.193200 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.193082 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-tls-certs\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.193200 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.193117 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4dtk\" (UniqueName: \"kubernetes.io/projected/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-kube-api-access-x4dtk\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.193277 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.193211 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-dshm\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.193277 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.193240 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-model-cache\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.193277 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.193274 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-tmp-dir\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.294319 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.294278 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-tmp-dir\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.294510 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.294337 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-home\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.294510 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.294380 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-tls-certs\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.294510 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.294415 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4dtk\" (UniqueName: \"kubernetes.io/projected/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-kube-api-access-x4dtk\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.294510 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.294480 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-dshm\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.294510 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.294507 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-model-cache\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.294789 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.294764 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-tmp-dir\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.294848 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.294785 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-home\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.294890 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.294868 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-model-cache\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.296849 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.296820 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-dshm\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.297116 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.297099 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-tls-certs\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.304057 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.304031 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4dtk\" (UniqueName: \"kubernetes.io/projected/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-kube-api-access-x4dtk\") pod \"auth-test-model-0af48d-kserve-7c78886f59-4msds\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.375959 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.375923 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:36.521140 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.521116 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds"] Jul 10 18:16:36.796986 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.796884 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" event={"ID":"1eaf45c6-0e67-40b6-98ef-ae444e23ab33","Type":"ContainerStarted","Data":"34462c588111a03c361bc718003a0f7393ab48c6b6fc5c3ac6d6918514b3c344"} Jul 10 18:16:36.796986 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.796931 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" event={"ID":"1eaf45c6-0e67-40b6-98ef-ae444e23ab33","Type":"ContainerStarted","Data":"59ffbba7fd01372489af2b9920d07b954f475db6c6fefa93c2a085547d4d7886"} Jul 10 18:16:36.815140 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:36.815077 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" podStartSLOduration=0.815058767 podStartE2EDuration="815.058767ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:16:36.81470982 +0000 UTC m=+2246.089036515" watchObservedRunningTime="2026-07-10 18:16:36.815058767 +0000 UTC m=+2246.089385493" Jul 10 18:16:39.872895 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:39.872858 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:16:46.376333 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:46.376299 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:46.376733 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:46.376359 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:46.389030 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:46.388996 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:46.856219 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:46.856194 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:16:53.552268 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:16:53.552235 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:17:12.161693 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:17:12.161655 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:17:29.657257 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:17:29.657216 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:17:48.066867 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:17:48.066833 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:18:05.868752 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:05.868674 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:18:24.460638 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:24.460600 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:18:39.141080 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:39.141045 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh"] Jul 10 18:18:40.043091 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:40.043060 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds"] Jul 10 18:18:44.357073 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:44.357037 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:18:54.162053 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.162009 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" podUID="edcef346-3dd0-4941-bc54-ae54857706e8" containerName="main" containerID="cri-o://3e3f6c8abbbfc285060ef68cdf97ee2fc456b3d486ef858f69f3ddeae1599fee" gracePeriod=45 Jul 10 18:18:54.447066 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.447041 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:18:54.477841 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.477805 2556 generic.go:356] "Generic (PLEG): container finished" podID="edcef346-3dd0-4941-bc54-ae54857706e8" containerID="3e3f6c8abbbfc285060ef68cdf97ee2fc456b3d486ef858f69f3ddeae1599fee" exitCode=0 Jul 10 18:18:54.478056 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.477863 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" Jul 10 18:18:54.478056 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.477890 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" event={"ID":"edcef346-3dd0-4941-bc54-ae54857706e8","Type":"ContainerDied","Data":"3e3f6c8abbbfc285060ef68cdf97ee2fc456b3d486ef858f69f3ddeae1599fee"} Jul 10 18:18:54.478056 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.477933 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh" event={"ID":"edcef346-3dd0-4941-bc54-ae54857706e8","Type":"ContainerDied","Data":"01e7885ac8220ebe1d9c45a4c3b1fc82a35e1f7558b2cdc9bc5269bb08cdc5e8"} Jul 10 18:18:54.478056 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.477953 2556 scope.go:117] "RemoveContainer" containerID="3e3f6c8abbbfc285060ef68cdf97ee2fc456b3d486ef858f69f3ddeae1599fee" Jul 10 18:18:54.486683 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.486652 2556 scope.go:117] "RemoveContainer" containerID="3e3f6c8abbbfc285060ef68cdf97ee2fc456b3d486ef858f69f3ddeae1599fee" Jul 10 18:18:54.487003 ip-10-0-137-103 kubenswrapper[2556]: E0710 18:18:54.486981 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3f6c8abbbfc285060ef68cdf97ee2fc456b3d486ef858f69f3ddeae1599fee\": container with ID starting with 3e3f6c8abbbfc285060ef68cdf97ee2fc456b3d486ef858f69f3ddeae1599fee not found: ID does not exist" containerID="3e3f6c8abbbfc285060ef68cdf97ee2fc456b3d486ef858f69f3ddeae1599fee" Jul 10 18:18:54.487076 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.487013 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3f6c8abbbfc285060ef68cdf97ee2fc456b3d486ef858f69f3ddeae1599fee"} err="failed to get container status \"3e3f6c8abbbfc285060ef68cdf97ee2fc456b3d486ef858f69f3ddeae1599fee\": rpc error: code = NotFound desc = could not find container \"3e3f6c8abbbfc285060ef68cdf97ee2fc456b3d486ef858f69f3ddeae1599fee\": container with ID starting with 3e3f6c8abbbfc285060ef68cdf97ee2fc456b3d486ef858f69f3ddeae1599fee not found: ID does not exist" Jul 10 18:18:54.595057 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.595017 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjbt2\" (UniqueName: \"kubernetes.io/projected/edcef346-3dd0-4941-bc54-ae54857706e8-kube-api-access-gjbt2\") pod \"edcef346-3dd0-4941-bc54-ae54857706e8\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " Jul 10 18:18:54.595057 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.595059 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-home\") pod \"edcef346-3dd0-4941-bc54-ae54857706e8\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " Jul 10 18:18:54.595291 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.595093 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-dshm\") pod \"edcef346-3dd0-4941-bc54-ae54857706e8\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " Jul 10 18:18:54.595291 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.595121 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-tmp-dir\") pod \"edcef346-3dd0-4941-bc54-ae54857706e8\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " Jul 10 18:18:54.595291 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.595181 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edcef346-3dd0-4941-bc54-ae54857706e8-tls-certs\") pod \"edcef346-3dd0-4941-bc54-ae54857706e8\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " Jul 10 18:18:54.595291 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.595203 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-model-cache\") pod \"edcef346-3dd0-4941-bc54-ae54857706e8\" (UID: \"edcef346-3dd0-4941-bc54-ae54857706e8\") " Jul 10 18:18:54.595519 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.595461 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-home" (OuterVolumeSpecName: "home") pod "edcef346-3dd0-4941-bc54-ae54857706e8" (UID: "edcef346-3dd0-4941-bc54-ae54857706e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:18:54.595519 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.595475 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "edcef346-3dd0-4941-bc54-ae54857706e8" (UID: "edcef346-3dd0-4941-bc54-ae54857706e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:18:54.595519 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.595503 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "edcef346-3dd0-4941-bc54-ae54857706e8" (UID: "edcef346-3dd0-4941-bc54-ae54857706e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:18:54.597584 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.597555 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-dshm" (OuterVolumeSpecName: "dshm") pod "edcef346-3dd0-4941-bc54-ae54857706e8" (UID: "edcef346-3dd0-4941-bc54-ae54857706e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:18:54.597692 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.597590 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edcef346-3dd0-4941-bc54-ae54857706e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edcef346-3dd0-4941-bc54-ae54857706e8" (UID: "edcef346-3dd0-4941-bc54-ae54857706e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:18:54.597692 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.597671 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcef346-3dd0-4941-bc54-ae54857706e8-kube-api-access-gjbt2" (OuterVolumeSpecName: "kube-api-access-gjbt2") pod "edcef346-3dd0-4941-bc54-ae54857706e8" (UID: "edcef346-3dd0-4941-bc54-ae54857706e8"). InnerVolumeSpecName "kube-api-access-gjbt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:18:54.696220 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.696118 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-dshm\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:18:54.696220 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.696178 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-tmp-dir\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:18:54.696220 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.696193 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edcef346-3dd0-4941-bc54-ae54857706e8-tls-certs\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:18:54.696220 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.696204 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-model-cache\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:18:54.696220 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.696217 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjbt2\" (UniqueName: \"kubernetes.io/projected/edcef346-3dd0-4941-bc54-ae54857706e8-kube-api-access-gjbt2\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:18:54.696497 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.696230 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edcef346-3dd0-4941-bc54-ae54857706e8-home\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:18:54.802665 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.802628 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh"] Jul 10 18:18:54.808455 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:54.808427 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/auth-test-model-b8bbaf-kserve-7d45dffcc5-fkvnh"] Jul 10 18:18:55.064749 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.064652 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" podUID="1eaf45c6-0e67-40b6-98ef-ae444e23ab33" containerName="main" containerID="cri-o://34462c588111a03c361bc718003a0f7393ab48c6b6fc5c3ac6d6918514b3c344" gracePeriod=45 Jul 10 18:18:55.258536 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.258500 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edcef346-3dd0-4941-bc54-ae54857706e8" path="/var/lib/kubelet/pods/edcef346-3dd0-4941-bc54-ae54857706e8/volumes" Jul 10 18:18:55.318854 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.318786 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:18:55.485256 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.485224 2556 generic.go:356] "Generic (PLEG): container finished" podID="1eaf45c6-0e67-40b6-98ef-ae444e23ab33" containerID="34462c588111a03c361bc718003a0f7393ab48c6b6fc5c3ac6d6918514b3c344" exitCode=0 Jul 10 18:18:55.485431 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.485264 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" event={"ID":"1eaf45c6-0e67-40b6-98ef-ae444e23ab33","Type":"ContainerDied","Data":"34462c588111a03c361bc718003a0f7393ab48c6b6fc5c3ac6d6918514b3c344"} Jul 10 18:18:55.485431 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.485269 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" Jul 10 18:18:55.485431 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.485294 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds" event={"ID":"1eaf45c6-0e67-40b6-98ef-ae444e23ab33","Type":"ContainerDied","Data":"59ffbba7fd01372489af2b9920d07b954f475db6c6fefa93c2a085547d4d7886"} Jul 10 18:18:55.485431 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.485312 2556 scope.go:117] "RemoveContainer" containerID="34462c588111a03c361bc718003a0f7393ab48c6b6fc5c3ac6d6918514b3c344" Jul 10 18:18:55.495880 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.495860 2556 scope.go:117] "RemoveContainer" containerID="34462c588111a03c361bc718003a0f7393ab48c6b6fc5c3ac6d6918514b3c344" Jul 10 18:18:55.496188 ip-10-0-137-103 kubenswrapper[2556]: E0710 18:18:55.496169 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34462c588111a03c361bc718003a0f7393ab48c6b6fc5c3ac6d6918514b3c344\": container with ID starting with 34462c588111a03c361bc718003a0f7393ab48c6b6fc5c3ac6d6918514b3c344 not found: ID does not exist" containerID="34462c588111a03c361bc718003a0f7393ab48c6b6fc5c3ac6d6918514b3c344" Jul 10 18:18:55.496268 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.496201 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34462c588111a03c361bc718003a0f7393ab48c6b6fc5c3ac6d6918514b3c344"} err="failed to get container status \"34462c588111a03c361bc718003a0f7393ab48c6b6fc5c3ac6d6918514b3c344\": rpc error: code = NotFound desc = could not find container \"34462c588111a03c361bc718003a0f7393ab48c6b6fc5c3ac6d6918514b3c344\": container with ID starting with 34462c588111a03c361bc718003a0f7393ab48c6b6fc5c3ac6d6918514b3c344 not found: ID does not exist" Jul 10 18:18:55.503511 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.503487 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-home\") pod \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " Jul 10 18:18:55.503594 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.503537 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-tls-certs\") pod \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " Jul 10 18:18:55.503656 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.503598 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4dtk\" (UniqueName: \"kubernetes.io/projected/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-kube-api-access-x4dtk\") pod \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " Jul 10 18:18:55.503656 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.503648 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-tmp-dir\") pod \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " Jul 10 18:18:55.503761 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.503686 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-dshm\") pod \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " Jul 10 18:18:55.503820 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.503770 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-model-cache\") pod \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\" (UID: \"1eaf45c6-0e67-40b6-98ef-ae444e23ab33\") " Jul 10 18:18:55.503820 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.503782 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-home" (OuterVolumeSpecName: "home") pod "1eaf45c6-0e67-40b6-98ef-ae444e23ab33" (UID: "1eaf45c6-0e67-40b6-98ef-ae444e23ab33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:18:55.504010 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.503982 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1eaf45c6-0e67-40b6-98ef-ae444e23ab33" (UID: "1eaf45c6-0e67-40b6-98ef-ae444e23ab33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:18:55.504134 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.504111 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-home\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:18:55.504193 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.504164 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-tmp-dir\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:18:55.504246 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.504201 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-model-cache" (OuterVolumeSpecName: "model-cache") pod "1eaf45c6-0e67-40b6-98ef-ae444e23ab33" (UID: "1eaf45c6-0e67-40b6-98ef-ae444e23ab33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:18:55.506115 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.506088 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-dshm" (OuterVolumeSpecName: "dshm") pod "1eaf45c6-0e67-40b6-98ef-ae444e23ab33" (UID: "1eaf45c6-0e67-40b6-98ef-ae444e23ab33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:18:55.506214 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.506182 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1eaf45c6-0e67-40b6-98ef-ae444e23ab33" (UID: "1eaf45c6-0e67-40b6-98ef-ae444e23ab33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:18:55.506214 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.506192 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-kube-api-access-x4dtk" (OuterVolumeSpecName: "kube-api-access-x4dtk") pod "1eaf45c6-0e67-40b6-98ef-ae444e23ab33" (UID: "1eaf45c6-0e67-40b6-98ef-ae444e23ab33"). InnerVolumeSpecName "kube-api-access-x4dtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:18:55.604891 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.604864 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4dtk\" (UniqueName: \"kubernetes.io/projected/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-kube-api-access-x4dtk\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:18:55.604891 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.604890 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-dshm\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:18:55.605117 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.604905 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-model-cache\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:18:55.605117 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.604914 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eaf45c6-0e67-40b6-98ef-ae444e23ab33-tls-certs\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:18:55.806927 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.806894 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds"] Jul 10 18:18:55.810451 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:55.810421 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/auth-test-model-0af48d-kserve-7c78886f59-4msds"] Jul 10 18:18:57.255800 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:18:57.255762 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eaf45c6-0e67-40b6-98ef-ae444e23ab33" path="/var/lib/kubelet/pods/1eaf45c6-0e67-40b6-98ef-ae444e23ab33/volumes" Jul 10 18:19:00.152895 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:00.152856 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:19:11.313839 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:11.313807 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 18:19:11.317540 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:11.317514 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 18:19:31.461261 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.461172 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx"] Jul 10 18:19:31.461743 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.461576 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edcef346-3dd0-4941-bc54-ae54857706e8" containerName="main" Jul 10 18:19:31.461743 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.461590 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="edcef346-3dd0-4941-bc54-ae54857706e8" containerName="main" Jul 10 18:19:31.461743 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.461599 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eaf45c6-0e67-40b6-98ef-ae444e23ab33" containerName="main" Jul 10 18:19:31.461743 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.461604 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eaf45c6-0e67-40b6-98ef-ae444e23ab33" containerName="main" Jul 10 18:19:31.461743 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.461669 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="edcef346-3dd0-4941-bc54-ae54857706e8" containerName="main" Jul 10 18:19:31.461743 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.461676 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eaf45c6-0e67-40b6-98ef-ae444e23ab33" containerName="main" Jul 10 18:19:31.464348 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.464332 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.468000 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.467953 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0af48d\"/\"kube-root-ca.crt\"" Jul 10 18:19:31.468139 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.467953 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0af48d\"/\"default-dockercfg-jdhgv\"" Jul 10 18:19:31.468139 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.467955 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0af48d\"/\"sub-test-model-0af48d-kserve-self-signed-certs\"" Jul 10 18:19:31.468139 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.467955 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0af48d\"/\"openshift-service-ca.crt\"" Jul 10 18:19:31.474829 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.474806 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx"] Jul 10 18:19:31.540028 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.539992 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-tmp-dir\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.540213 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.540039 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe50420-0cb5-4654-8930-520e1a8ed186-tls-certs\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.540213 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.540067 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-home\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.540213 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.540122 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-dshm\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.540213 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.540193 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-model-cache\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.540354 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.540218 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7gp6\" (UniqueName: \"kubernetes.io/projected/4fe50420-0cb5-4654-8930-520e1a8ed186-kube-api-access-g7gp6\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.641266 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.641217 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-tmp-dir\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.641470 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.641279 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe50420-0cb5-4654-8930-520e1a8ed186-tls-certs\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.641470 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.641326 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-home\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.641470 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.641346 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-dshm\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.641470 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.641391 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-model-cache\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.641470 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.641421 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7gp6\" (UniqueName: \"kubernetes.io/projected/4fe50420-0cb5-4654-8930-520e1a8ed186-kube-api-access-g7gp6\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.641676 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.641658 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-tmp-dir\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.641721 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.641703 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-model-cache\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.641757 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.641728 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-home\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.643719 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.643696 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-dshm\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.644037 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.644017 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe50420-0cb5-4654-8930-520e1a8ed186-tls-certs\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.650327 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.650301 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7gp6\" (UniqueName: \"kubernetes.io/projected/4fe50420-0cb5-4654-8930-520e1a8ed186-kube-api-access-g7gp6\") pod \"sub-test-model-0af48d-kserve-6494547fc-cq9jx\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.773943 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.773847 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:31.919141 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:31.919111 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx"] Jul 10 18:19:31.921030 ip-10-0-137-103 kubenswrapper[2556]: W0710 18:19:31.921000 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fe50420_0cb5_4654_8930_520e1a8ed186.slice/crio-2de2b335146287de51bf053759647dfcbd2731add0d59b8c0caf75ec498316a4 WatchSource:0}: Error finding container 2de2b335146287de51bf053759647dfcbd2731add0d59b8c0caf75ec498316a4: Status 404 returned error can't find the container with id 2de2b335146287de51bf053759647dfcbd2731add0d59b8c0caf75ec498316a4 Jul 10 18:19:32.670334 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:32.670295 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" event={"ID":"4fe50420-0cb5-4654-8930-520e1a8ed186","Type":"ContainerStarted","Data":"a61de0afe9d2d0856d7662bc8bd9b405e01eefbc3995a0ea4a9ec0e6ddaa45de"} Jul 10 18:19:32.670708 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:32.670347 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" event={"ID":"4fe50420-0cb5-4654-8930-520e1a8ed186","Type":"ContainerStarted","Data":"2de2b335146287de51bf053759647dfcbd2731add0d59b8c0caf75ec498316a4"} Jul 10 18:19:32.688149 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:32.688099 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" podStartSLOduration=1.688081208 podStartE2EDuration="1.688081208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:19:32.685962833 +0000 UTC m=+2421.960289543" watchObservedRunningTime="2026-07-10 18:19:32.688081208 +0000 UTC m=+2421.962407902" Jul 10 18:19:35.656086 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:35.656023 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:19:41.774065 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:41.774027 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:41.774627 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:41.774076 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:41.788650 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:41.788619 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:19:42.731684 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:19:42.731647 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:20:01.059881 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:01.059838 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:20:16.939873 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:16.939840 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx"] Jul 10 18:20:20.063781 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.063746 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:20:20.553034 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.552999 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5"] Jul 10 18:20:20.556448 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.556429 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.558844 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.558823 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b8bbaf\"/\"kube-root-ca.crt\"" Jul 10 18:20:20.559854 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.559839 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b8bbaf\"/\"openshift-service-ca.crt\"" Jul 10 18:20:20.559980 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.559949 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b8bbaf\"/\"default-dockercfg-sdccd\"" Jul 10 18:20:20.560037 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.560017 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b8bbaf\"/\"rate-test-model-b8bbaf-kserve-self-signed-certs\"" Jul 10 18:20:20.565938 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.565912 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5"] Jul 10 18:20:20.695766 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.695734 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-home\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.695766 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.695774 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-tmp-dir\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.696022 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.695810 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-model-cache\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.696022 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.695917 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv7p8\" (UniqueName: \"kubernetes.io/projected/09161c47-6950-485a-87df-417a3a68130e-kube-api-access-qv7p8\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.696022 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.695948 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-dshm\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.696022 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.696019 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09161c47-6950-485a-87df-417a3a68130e-tls-certs\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.796805 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.796766 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-model-cache\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.797018 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.796834 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv7p8\" (UniqueName: \"kubernetes.io/projected/09161c47-6950-485a-87df-417a3a68130e-kube-api-access-qv7p8\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.797018 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.796856 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-dshm\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.797116 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.797019 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09161c47-6950-485a-87df-417a3a68130e-tls-certs\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.797116 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.797097 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-home\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.797214 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.797141 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-tmp-dir\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.797269 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.797241 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-model-cache\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.797457 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.797435 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-home\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.797531 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.797459 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-tmp-dir\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.799280 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.799251 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-dshm\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.799567 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.799549 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09161c47-6950-485a-87df-417a3a68130e-tls-certs\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.805448 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.805400 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv7p8\" (UniqueName: \"kubernetes.io/projected/09161c47-6950-485a-87df-417a3a68130e-kube-api-access-qv7p8\") pod \"rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:20.866608 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:20.866572 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:21.004670 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:21.004634 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5"] Jul 10 18:20:21.007944 ip-10-0-137-103 kubenswrapper[2556]: W0710 18:20:21.007913 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09161c47_6950_485a_87df_417a3a68130e.slice/crio-5d94efca7adde3df904d833a8f46c5a899a95719722ae7931942d92dbc462432 WatchSource:0}: Error finding container 5d94efca7adde3df904d833a8f46c5a899a95719722ae7931942d92dbc462432: Status 404 returned error can't find the container with id 5d94efca7adde3df904d833a8f46c5a899a95719722ae7931942d92dbc462432 Jul 10 18:20:21.917620 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:21.917582 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" event={"ID":"09161c47-6950-485a-87df-417a3a68130e","Type":"ContainerStarted","Data":"0b305e7d30c9fc26ddb9b85a270946e93547b9b4c2877df35f5ad69786ee4ab9"} Jul 10 18:20:21.917620 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:21.917624 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" event={"ID":"09161c47-6950-485a-87df-417a3a68130e","Type":"ContainerStarted","Data":"5d94efca7adde3df904d833a8f46c5a899a95719722ae7931942d92dbc462432"} Jul 10 18:20:21.934067 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:21.934009 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" podStartSLOduration=1.933992532 podStartE2EDuration="1.933992532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:20:21.933131191 +0000 UTC m=+2471.207457887" watchObservedRunningTime="2026-07-10 18:20:21.933992532 +0000 UTC m=+2471.208319223" Jul 10 18:20:30.867089 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:30.867049 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:30.867530 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:30.867105 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:30.879662 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:30.879633 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:30.972599 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:30.972572 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:20:31.963059 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:31.962994 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" podUID="4fe50420-0cb5-4654-8930-520e1a8ed186" containerName="main" containerID="cri-o://a61de0afe9d2d0856d7662bc8bd9b405e01eefbc3995a0ea4a9ec0e6ddaa45de" gracePeriod=45 Jul 10 18:20:32.214346 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.214283 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:20:32.308610 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.308576 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe50420-0cb5-4654-8930-520e1a8ed186-tls-certs\") pod \"4fe50420-0cb5-4654-8930-520e1a8ed186\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " Jul 10 18:20:32.308610 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.308614 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-dshm\") pod \"4fe50420-0cb5-4654-8930-520e1a8ed186\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " Jul 10 18:20:32.308879 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.308660 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-model-cache\") pod \"4fe50420-0cb5-4654-8930-520e1a8ed186\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " Jul 10 18:20:32.308879 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.308738 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7gp6\" (UniqueName: \"kubernetes.io/projected/4fe50420-0cb5-4654-8930-520e1a8ed186-kube-api-access-g7gp6\") pod \"4fe50420-0cb5-4654-8930-520e1a8ed186\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " Jul 10 18:20:32.308879 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.308830 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-home\") pod \"4fe50420-0cb5-4654-8930-520e1a8ed186\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " Jul 10 18:20:32.309058 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.308881 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-tmp-dir\") pod \"4fe50420-0cb5-4654-8930-520e1a8ed186\" (UID: \"4fe50420-0cb5-4654-8930-520e1a8ed186\") " Jul 10 18:20:32.309058 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.309035 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fe50420-0cb5-4654-8930-520e1a8ed186" (UID: "4fe50420-0cb5-4654-8930-520e1a8ed186"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:20:32.309186 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.309155 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-home" (OuterVolumeSpecName: "home") pod "4fe50420-0cb5-4654-8930-520e1a8ed186" (UID: "4fe50420-0cb5-4654-8930-520e1a8ed186"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:20:32.309319 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.309186 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fe50420-0cb5-4654-8930-520e1a8ed186" (UID: "4fe50420-0cb5-4654-8930-520e1a8ed186"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:20:32.309550 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.309376 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-model-cache\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:20:32.309550 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.309496 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-home\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:20:32.309550 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.309510 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-tmp-dir\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:20:32.310983 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.310945 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-dshm" (OuterVolumeSpecName: "dshm") pod "4fe50420-0cb5-4654-8930-520e1a8ed186" (UID: "4fe50420-0cb5-4654-8930-520e1a8ed186"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:20:32.311088 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.311027 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe50420-0cb5-4654-8930-520e1a8ed186-kube-api-access-g7gp6" (OuterVolumeSpecName: "kube-api-access-g7gp6") pod "4fe50420-0cb5-4654-8930-520e1a8ed186" (UID: "4fe50420-0cb5-4654-8930-520e1a8ed186"). InnerVolumeSpecName "kube-api-access-g7gp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:20:32.311088 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.311078 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe50420-0cb5-4654-8930-520e1a8ed186-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fe50420-0cb5-4654-8930-520e1a8ed186" (UID: "4fe50420-0cb5-4654-8930-520e1a8ed186"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:20:32.409995 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.409936 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7gp6\" (UniqueName: \"kubernetes.io/projected/4fe50420-0cb5-4654-8930-520e1a8ed186-kube-api-access-g7gp6\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:20:32.409995 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.409990 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe50420-0cb5-4654-8930-520e1a8ed186-tls-certs\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:20:32.409995 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.410004 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe50420-0cb5-4654-8930-520e1a8ed186-dshm\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:20:32.972158 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.972123 2556 generic.go:356] "Generic (PLEG): container finished" podID="4fe50420-0cb5-4654-8930-520e1a8ed186" containerID="a61de0afe9d2d0856d7662bc8bd9b405e01eefbc3995a0ea4a9ec0e6ddaa45de" exitCode=0 Jul 10 18:20:32.972569 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.972170 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" Jul 10 18:20:32.972569 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.972204 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" event={"ID":"4fe50420-0cb5-4654-8930-520e1a8ed186","Type":"ContainerDied","Data":"a61de0afe9d2d0856d7662bc8bd9b405e01eefbc3995a0ea4a9ec0e6ddaa45de"} Jul 10 18:20:32.972569 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.972250 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx" event={"ID":"4fe50420-0cb5-4654-8930-520e1a8ed186","Type":"ContainerDied","Data":"2de2b335146287de51bf053759647dfcbd2731add0d59b8c0caf75ec498316a4"} Jul 10 18:20:32.972569 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.972268 2556 scope.go:117] "RemoveContainer" containerID="a61de0afe9d2d0856d7662bc8bd9b405e01eefbc3995a0ea4a9ec0e6ddaa45de" Jul 10 18:20:32.981720 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.981700 2556 scope.go:117] "RemoveContainer" containerID="a61de0afe9d2d0856d7662bc8bd9b405e01eefbc3995a0ea4a9ec0e6ddaa45de" Jul 10 18:20:32.981994 ip-10-0-137-103 kubenswrapper[2556]: E0710 18:20:32.981956 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61de0afe9d2d0856d7662bc8bd9b405e01eefbc3995a0ea4a9ec0e6ddaa45de\": container with ID starting with a61de0afe9d2d0856d7662bc8bd9b405e01eefbc3995a0ea4a9ec0e6ddaa45de not found: ID does not exist" containerID="a61de0afe9d2d0856d7662bc8bd9b405e01eefbc3995a0ea4a9ec0e6ddaa45de" Jul 10 18:20:32.982100 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.981999 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61de0afe9d2d0856d7662bc8bd9b405e01eefbc3995a0ea4a9ec0e6ddaa45de"} err="failed to get container status \"a61de0afe9d2d0856d7662bc8bd9b405e01eefbc3995a0ea4a9ec0e6ddaa45de\": rpc error: code = NotFound desc = could not find container \"a61de0afe9d2d0856d7662bc8bd9b405e01eefbc3995a0ea4a9ec0e6ddaa45de\": container with ID starting with a61de0afe9d2d0856d7662bc8bd9b405e01eefbc3995a0ea4a9ec0e6ddaa45de not found: ID does not exist" Jul 10 18:20:32.992829 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.992803 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx"] Jul 10 18:20:32.997812 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:32.997787 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/sub-test-model-0af48d-kserve-6494547fc-cq9jx"] Jul 10 18:20:33.254931 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:33.254856 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fe50420-0cb5-4654-8930-520e1a8ed186" path="/var/lib/kubelet/pods/4fe50420-0cb5-4654-8930-520e1a8ed186/volumes" Jul 10 18:20:36.955339 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:20:36.955300 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:21:16.852558 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:21:16.852519 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:21:48.847827 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:21:48.847787 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:22:02.754738 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:02.754695 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:22:09.102810 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:09.102771 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5"] Jul 10 18:22:11.083098 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:11.083058 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7406-140152214527536"] Jul 10 18:22:11.083640 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:11.083620 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe50420-0cb5-4654-8930-520e1a8ed186" containerName="main" Jul 10 18:22:11.083721 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:11.083642 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe50420-0cb5-4654-8930-520e1a8ed186" containerName="main" Jul 10 18:22:11.083772 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:11.083732 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe50420-0cb5-4654-8930-520e1a8ed186" containerName="main" Jul 10 18:22:11.086836 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:11.086814 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152214527536" Jul 10 18:22:11.089533 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:11.089507 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-cdwmd\"" Jul 10 18:22:11.090913 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:11.090888 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7406-140152214527536"] Jul 10 18:22:11.130770 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:11.130730 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d2cr\" (UniqueName: \"kubernetes.io/projected/58d59be2-4d9b-4fb2-9b0a-60c23001bb15-kube-api-access-5d2cr\") pod \"test-curl-7406-140152214527536\" (UID: \"58d59be2-4d9b-4fb2-9b0a-60c23001bb15\") " pod="openshift-ingress/test-curl-7406-140152214527536" Jul 10 18:22:11.231891 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:11.231855 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d2cr\" (UniqueName: \"kubernetes.io/projected/58d59be2-4d9b-4fb2-9b0a-60c23001bb15-kube-api-access-5d2cr\") pod \"test-curl-7406-140152214527536\" (UID: \"58d59be2-4d9b-4fb2-9b0a-60c23001bb15\") " pod="openshift-ingress/test-curl-7406-140152214527536" Jul 10 18:22:11.240817 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:11.240785 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d2cr\" (UniqueName: \"kubernetes.io/projected/58d59be2-4d9b-4fb2-9b0a-60c23001bb15-kube-api-access-5d2cr\") pod \"test-curl-7406-140152214527536\" (UID: \"58d59be2-4d9b-4fb2-9b0a-60c23001bb15\") " pod="openshift-ingress/test-curl-7406-140152214527536" Jul 10 18:22:11.397135 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:11.397102 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152214527536" Jul 10 18:22:11.533958 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:11.533933 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7406-140152214527536"] Jul 10 18:22:11.535929 ip-10-0-137-103 kubenswrapper[2556]: W0710 18:22:11.535903 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58d59be2_4d9b_4fb2_9b0a_60c23001bb15.slice/crio-a62c9c856aa6aec7d99bd8a2799396a1bdbb34725cc6741bbdbbaaaf1f35ec5e WatchSource:0}: Error finding container a62c9c856aa6aec7d99bd8a2799396a1bdbb34725cc6741bbdbbaaaf1f35ec5e: Status 404 returned error can't find the container with id a62c9c856aa6aec7d99bd8a2799396a1bdbb34725cc6741bbdbbaaaf1f35ec5e Jul 10 18:22:11.537362 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:11.537341 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:22:12.466808 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:12.466768 2556 generic.go:356] "Generic (PLEG): container finished" podID="58d59be2-4d9b-4fb2-9b0a-60c23001bb15" containerID="7e5f0a00fd5eb8ab4204693c3c911336f9f990eff412eedd3593e24b57fc7662" exitCode=0 Jul 10 18:22:12.467243 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:12.466851 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7406-140152214527536" event={"ID":"58d59be2-4d9b-4fb2-9b0a-60c23001bb15","Type":"ContainerDied","Data":"7e5f0a00fd5eb8ab4204693c3c911336f9f990eff412eedd3593e24b57fc7662"} Jul 10 18:22:12.467243 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:12.466900 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7406-140152214527536" event={"ID":"58d59be2-4d9b-4fb2-9b0a-60c23001bb15","Type":"ContainerStarted","Data":"a62c9c856aa6aec7d99bd8a2799396a1bdbb34725cc6741bbdbbaaaf1f35ec5e"} Jul 10 18:22:13.624305 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.624273 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152214527536" Jul 10 18:22:13.657278 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.657246 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7406-140152214527536_58d59be2-4d9b-4fb2-9b0a-60c23001bb15/test-curl-7406-140152214527536/0.log" Jul 10 18:22:13.678923 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.678886 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7406-140152214527536"] Jul 10 18:22:13.681086 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.681060 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7406-140152214527536"] Jul 10 18:22:13.754575 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.754476 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d2cr\" (UniqueName: \"kubernetes.io/projected/58d59be2-4d9b-4fb2-9b0a-60c23001bb15-kube-api-access-5d2cr\") pod \"58d59be2-4d9b-4fb2-9b0a-60c23001bb15\" (UID: \"58d59be2-4d9b-4fb2-9b0a-60c23001bb15\") " Jul 10 18:22:13.756787 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.756757 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d59be2-4d9b-4fb2-9b0a-60c23001bb15-kube-api-access-5d2cr" (OuterVolumeSpecName: "kube-api-access-5d2cr") pod "58d59be2-4d9b-4fb2-9b0a-60c23001bb15" (UID: "58d59be2-4d9b-4fb2-9b0a-60c23001bb15"). InnerVolumeSpecName "kube-api-access-5d2cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:22:13.797228 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.797191 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7406-140152214524080"] Jul 10 18:22:13.797572 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.797560 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d59be2-4d9b-4fb2-9b0a-60c23001bb15" containerName="test-curl-7406-140152214527536" Jul 10 18:22:13.797616 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.797574 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d59be2-4d9b-4fb2-9b0a-60c23001bb15" containerName="test-curl-7406-140152214527536" Jul 10 18:22:13.797659 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.797650 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d59be2-4d9b-4fb2-9b0a-60c23001bb15" containerName="test-curl-7406-140152214527536" Jul 10 18:22:13.800788 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.800769 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152214524080" Jul 10 18:22:13.806782 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.806750 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7406-140152214524080"] Jul 10 18:22:13.855765 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.855726 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4425k\" (UniqueName: \"kubernetes.io/projected/5aecbfb4-69f7-4287-97eb-484f7a1aec31-kube-api-access-4425k\") pod \"test-curl-7406-140152214524080\" (UID: \"5aecbfb4-69f7-4287-97eb-484f7a1aec31\") " pod="openshift-ingress/test-curl-7406-140152214524080" Jul 10 18:22:13.855948 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.855879 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d2cr\" (UniqueName: \"kubernetes.io/projected/58d59be2-4d9b-4fb2-9b0a-60c23001bb15-kube-api-access-5d2cr\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:22:13.957101 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.957060 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4425k\" (UniqueName: \"kubernetes.io/projected/5aecbfb4-69f7-4287-97eb-484f7a1aec31-kube-api-access-4425k\") pod \"test-curl-7406-140152214524080\" (UID: \"5aecbfb4-69f7-4287-97eb-484f7a1aec31\") " pod="openshift-ingress/test-curl-7406-140152214524080" Jul 10 18:22:13.966657 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:13.966622 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4425k\" (UniqueName: \"kubernetes.io/projected/5aecbfb4-69f7-4287-97eb-484f7a1aec31-kube-api-access-4425k\") pod \"test-curl-7406-140152214524080\" (UID: \"5aecbfb4-69f7-4287-97eb-484f7a1aec31\") " pod="openshift-ingress/test-curl-7406-140152214524080" Jul 10 18:22:14.112182 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:14.112147 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152214524080" Jul 10 18:22:14.247579 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:14.247526 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7406-140152214524080"] Jul 10 18:22:14.249714 ip-10-0-137-103 kubenswrapper[2556]: W0710 18:22:14.249690 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5aecbfb4_69f7_4287_97eb_484f7a1aec31.slice/crio-82f1ec9d6fb176954489ceb86016f348dad4803b78c0ec2b4d16aeb8a8b941e4 WatchSource:0}: Error finding container 82f1ec9d6fb176954489ceb86016f348dad4803b78c0ec2b4d16aeb8a8b941e4: Status 404 returned error can't find the container with id 82f1ec9d6fb176954489ceb86016f348dad4803b78c0ec2b4d16aeb8a8b941e4 Jul 10 18:22:14.478538 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:14.478459 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152214527536" Jul 10 18:22:14.478538 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:14.478474 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a62c9c856aa6aec7d99bd8a2799396a1bdbb34725cc6741bbdbbaaaf1f35ec5e" Jul 10 18:22:14.479885 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:14.479851 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7406-140152214524080" event={"ID":"5aecbfb4-69f7-4287-97eb-484f7a1aec31","Type":"ContainerStarted","Data":"82f1ec9d6fb176954489ceb86016f348dad4803b78c0ec2b4d16aeb8a8b941e4"} Jul 10 18:22:14.489524 ip-10-0-137-103 kubenswrapper[2556]: E0710 18:22:14.489492 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7406-140152214527536\" is forbidden: User \"system:node:ip-10-0-137-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-137-103.ec2.internal' and this object" podUID="58d59be2-4d9b-4fb2-9b0a-60c23001bb15" pod="openshift-ingress/test-curl-7406-140152214527536" Jul 10 18:22:15.255248 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:15.255215 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58d59be2-4d9b-4fb2-9b0a-60c23001bb15" path="/var/lib/kubelet/pods/58d59be2-4d9b-4fb2-9b0a-60c23001bb15/volumes" Jul 10 18:22:15.486218 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:15.486188 2556 generic.go:356] "Generic (PLEG): container finished" podID="5aecbfb4-69f7-4287-97eb-484f7a1aec31" containerID="b97d12cc28aff3e51649d7e17260f6ae405d6faa724be16f1a4147f20bf5cf98" exitCode=0 Jul 10 18:22:15.486398 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:15.486286 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7406-140152214524080" event={"ID":"5aecbfb4-69f7-4287-97eb-484f7a1aec31","Type":"ContainerDied","Data":"b97d12cc28aff3e51649d7e17260f6ae405d6faa724be16f1a4147f20bf5cf98"} Jul 10 18:22:16.629077 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.629052 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152214524080" Jul 10 18:22:16.649205 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.649180 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7406-140152214524080_5aecbfb4-69f7-4287-97eb-484f7a1aec31/test-curl-7406-140152214524080/0.log" Jul 10 18:22:16.670912 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.670878 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7406-140152214524080"] Jul 10 18:22:16.675485 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.675459 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7406-140152214524080"] Jul 10 18:22:16.681702 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.681679 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4425k\" (UniqueName: \"kubernetes.io/projected/5aecbfb4-69f7-4287-97eb-484f7a1aec31-kube-api-access-4425k\") pod \"5aecbfb4-69f7-4287-97eb-484f7a1aec31\" (UID: \"5aecbfb4-69f7-4287-97eb-484f7a1aec31\") " Jul 10 18:22:16.683889 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.683864 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aecbfb4-69f7-4287-97eb-484f7a1aec31-kube-api-access-4425k" (OuterVolumeSpecName: "kube-api-access-4425k") pod "5aecbfb4-69f7-4287-97eb-484f7a1aec31" (UID: "5aecbfb4-69f7-4287-97eb-484f7a1aec31"). InnerVolumeSpecName "kube-api-access-4425k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:22:16.783224 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.783142 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4425k\" (UniqueName: \"kubernetes.io/projected/5aecbfb4-69f7-4287-97eb-484f7a1aec31-kube-api-access-4425k\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:22:16.831379 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.831335 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7406-140152221476736"] Jul 10 18:22:16.831740 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.831725 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aecbfb4-69f7-4287-97eb-484f7a1aec31" containerName="test-curl-7406-140152214524080" Jul 10 18:22:16.831794 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.831742 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aecbfb4-69f7-4287-97eb-484f7a1aec31" containerName="test-curl-7406-140152214524080" Jul 10 18:22:16.831845 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.831835 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aecbfb4-69f7-4287-97eb-484f7a1aec31" containerName="test-curl-7406-140152214524080" Jul 10 18:22:16.834731 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.834715 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:16.841040 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.841018 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7406-140152221476736"] Jul 10 18:22:16.884512 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.884477 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l44sc\" (UniqueName: \"kubernetes.io/projected/5204440d-2a69-42d0-a701-2cb475514265-kube-api-access-l44sc\") pod \"test-curl-7406-140152221476736\" (UID: \"5204440d-2a69-42d0-a701-2cb475514265\") " pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:16.985798 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.985757 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l44sc\" (UniqueName: \"kubernetes.io/projected/5204440d-2a69-42d0-a701-2cb475514265-kube-api-access-l44sc\") pod \"test-curl-7406-140152221476736\" (UID: \"5204440d-2a69-42d0-a701-2cb475514265\") " pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:16.994640 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:16.994620 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l44sc\" (UniqueName: \"kubernetes.io/projected/5204440d-2a69-42d0-a701-2cb475514265-kube-api-access-l44sc\") pod \"test-curl-7406-140152221476736\" (UID: \"5204440d-2a69-42d0-a701-2cb475514265\") " pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:17.144980 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:17.144943 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:17.153594 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:17.153558 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:22:17.257281 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:17.257243 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aecbfb4-69f7-4287-97eb-484f7a1aec31" path="/var/lib/kubelet/pods/5aecbfb4-69f7-4287-97eb-484f7a1aec31/volumes" Jul 10 18:22:17.283949 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:17.283920 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7406-140152221476736"] Jul 10 18:22:17.287222 ip-10-0-137-103 kubenswrapper[2556]: W0710 18:22:17.287191 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5204440d_2a69_42d0_a701_2cb475514265.slice/crio-f9848338bd8b9b6309f8629b0070d4301031006a280f53c09663c2d8f85e652e WatchSource:0}: Error finding container f9848338bd8b9b6309f8629b0070d4301031006a280f53c09663c2d8f85e652e: Status 404 returned error can't find the container with id f9848338bd8b9b6309f8629b0070d4301031006a280f53c09663c2d8f85e652e Jul 10 18:22:17.501260 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:17.501167 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152214524080" Jul 10 18:22:17.501260 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:17.501169 2556 scope.go:117] "RemoveContainer" containerID="b97d12cc28aff3e51649d7e17260f6ae405d6faa724be16f1a4147f20bf5cf98" Jul 10 18:22:17.502832 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:17.502593 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7406-140152221476736" event={"ID":"5204440d-2a69-42d0-a701-2cb475514265","Type":"ContainerStarted","Data":"f9848338bd8b9b6309f8629b0070d4301031006a280f53c09663c2d8f85e652e"} Jul 10 18:22:18.511435 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:18.511397 2556 generic.go:356] "Generic (PLEG): container finished" podID="5204440d-2a69-42d0-a701-2cb475514265" containerID="fa35fe67aa4dc93239a497bfb7f34783963db320790c93759b9cb31e8b216a59" exitCode=0 Jul 10 18:22:18.511807 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:18.511446 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7406-140152221476736" event={"ID":"5204440d-2a69-42d0-a701-2cb475514265","Type":"ContainerDied","Data":"fa35fe67aa4dc93239a497bfb7f34783963db320790c93759b9cb31e8b216a59"} Jul 10 18:22:19.676949 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.676923 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:19.695262 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.695231 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7406-140152221476736_5204440d-2a69-42d0-a701-2cb475514265/test-curl-7406-140152221476736/0.log" Jul 10 18:22:19.717776 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.717746 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7406-140152221476736"] Jul 10 18:22:19.723301 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.723273 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7406-140152221476736"] Jul 10 18:22:19.817485 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.817401 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l44sc\" (UniqueName: \"kubernetes.io/projected/5204440d-2a69-42d0-a701-2cb475514265-kube-api-access-l44sc\") pod \"5204440d-2a69-42d0-a701-2cb475514265\" (UID: \"5204440d-2a69-42d0-a701-2cb475514265\") " Jul 10 18:22:19.819753 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.819720 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5204440d-2a69-42d0-a701-2cb475514265-kube-api-access-l44sc" (OuterVolumeSpecName: "kube-api-access-l44sc") pod "5204440d-2a69-42d0-a701-2cb475514265" (UID: "5204440d-2a69-42d0-a701-2cb475514265"). InnerVolumeSpecName "kube-api-access-l44sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:22:19.844385 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.844352 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7406-140152221476736"] Jul 10 18:22:19.844756 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.844744 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5204440d-2a69-42d0-a701-2cb475514265" containerName="test-curl-7406-140152221476736" Jul 10 18:22:19.844800 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.844758 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5204440d-2a69-42d0-a701-2cb475514265" containerName="test-curl-7406-140152221476736" Jul 10 18:22:19.844834 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.844819 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5204440d-2a69-42d0-a701-2cb475514265" containerName="test-curl-7406-140152221476736" Jul 10 18:22:19.847600 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.847584 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:19.852356 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.852327 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7406-140152221476736"] Jul 10 18:22:19.918779 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.918741 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgxsx\" (UniqueName: \"kubernetes.io/projected/df97d2dc-bd16-42aa-a67e-d2f750dc6283-kube-api-access-lgxsx\") pod \"test-curl-7406-140152221476736\" (UID: \"df97d2dc-bd16-42aa-a67e-d2f750dc6283\") " pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:19.918963 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:19.918807 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l44sc\" (UniqueName: \"kubernetes.io/projected/5204440d-2a69-42d0-a701-2cb475514265-kube-api-access-l44sc\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:22:20.019681 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:20.019642 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgxsx\" (UniqueName: \"kubernetes.io/projected/df97d2dc-bd16-42aa-a67e-d2f750dc6283-kube-api-access-lgxsx\") pod \"test-curl-7406-140152221476736\" (UID: \"df97d2dc-bd16-42aa-a67e-d2f750dc6283\") " pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:20.028538 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:20.028507 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgxsx\" (UniqueName: \"kubernetes.io/projected/df97d2dc-bd16-42aa-a67e-d2f750dc6283-kube-api-access-lgxsx\") pod \"test-curl-7406-140152221476736\" (UID: \"df97d2dc-bd16-42aa-a67e-d2f750dc6283\") " pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:20.157704 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:20.157668 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:20.292134 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:20.292109 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7406-140152221476736"] Jul 10 18:22:20.294327 ip-10-0-137-103 kubenswrapper[2556]: W0710 18:22:20.294300 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf97d2dc_bd16_42aa_a67e_d2f750dc6283.slice/crio-d578b6c409a2641090e49d5c99e80eb82ee36a01a9bc5daf3b2c53beb62f01db WatchSource:0}: Error finding container d578b6c409a2641090e49d5c99e80eb82ee36a01a9bc5daf3b2c53beb62f01db: Status 404 returned error can't find the container with id d578b6c409a2641090e49d5c99e80eb82ee36a01a9bc5daf3b2c53beb62f01db Jul 10 18:22:20.522313 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:20.522214 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7406-140152221476736" event={"ID":"df97d2dc-bd16-42aa-a67e-d2f750dc6283","Type":"ContainerStarted","Data":"d578b6c409a2641090e49d5c99e80eb82ee36a01a9bc5daf3b2c53beb62f01db"} Jul 10 18:22:20.523526 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:20.523507 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9848338bd8b9b6309f8629b0070d4301031006a280f53c09663c2d8f85e652e" Jul 10 18:22:20.523526 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:20.523521 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:20.535468 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:20.535421 2556 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7406-140152221476736" oldPodUID="5204440d-2a69-42d0-a701-2cb475514265" podUID="df97d2dc-bd16-42aa-a67e-d2f750dc6283" Jul 10 18:22:21.256209 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:21.256177 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5204440d-2a69-42d0-a701-2cb475514265" path="/var/lib/kubelet/pods/5204440d-2a69-42d0-a701-2cb475514265/volumes" Jul 10 18:22:21.530114 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:21.530018 2556 generic.go:356] "Generic (PLEG): container finished" podID="df97d2dc-bd16-42aa-a67e-d2f750dc6283" containerID="cbc04b07103b16d2ac34279d36701db30206e37d9555487605e1df727e8d5d64" exitCode=0 Jul 10 18:22:21.530262 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:21.530108 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7406-140152221476736" event={"ID":"df97d2dc-bd16-42aa-a67e-d2f750dc6283","Type":"ContainerDied","Data":"cbc04b07103b16d2ac34279d36701db30206e37d9555487605e1df727e8d5d64"} Jul 10 18:22:22.684082 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.684058 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:22.704579 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.704542 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7406-140152221476736_df97d2dc-bd16-42aa-a67e-d2f750dc6283/test-curl-7406-140152221476736/0.log" Jul 10 18:22:22.726460 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.726422 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7406-140152221476736"] Jul 10 18:22:22.728382 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.728355 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7406-140152221476736"] Jul 10 18:22:22.849797 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.849715 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7406-140152221477744"] Jul 10 18:22:22.849947 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.849889 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgxsx\" (UniqueName: \"kubernetes.io/projected/df97d2dc-bd16-42aa-a67e-d2f750dc6283-kube-api-access-lgxsx\") pod \"df97d2dc-bd16-42aa-a67e-d2f750dc6283\" (UID: \"df97d2dc-bd16-42aa-a67e-d2f750dc6283\") " Jul 10 18:22:22.850141 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.850128 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df97d2dc-bd16-42aa-a67e-d2f750dc6283" containerName="test-curl-7406-140152221476736" Jul 10 18:22:22.850192 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.850142 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="df97d2dc-bd16-42aa-a67e-d2f750dc6283" containerName="test-curl-7406-140152221476736" Jul 10 18:22:22.850228 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.850207 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="df97d2dc-bd16-42aa-a67e-d2f750dc6283" containerName="test-curl-7406-140152221476736" Jul 10 18:22:22.852263 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.852233 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df97d2dc-bd16-42aa-a67e-d2f750dc6283-kube-api-access-lgxsx" (OuterVolumeSpecName: "kube-api-access-lgxsx") pod "df97d2dc-bd16-42aa-a67e-d2f750dc6283" (UID: "df97d2dc-bd16-42aa-a67e-d2f750dc6283"). InnerVolumeSpecName "kube-api-access-lgxsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:22:22.853334 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.853319 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:22.859169 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.859141 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7406-140152221477744"] Jul 10 18:22:22.951111 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.951080 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2ttd\" (UniqueName: \"kubernetes.io/projected/cd1e671b-021c-428f-830c-9cf66ebb19c2-kube-api-access-x2ttd\") pod \"test-curl-7406-140152221477744\" (UID: \"cd1e671b-021c-428f-830c-9cf66ebb19c2\") " pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:22.951295 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:22.951197 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgxsx\" (UniqueName: \"kubernetes.io/projected/df97d2dc-bd16-42aa-a67e-d2f750dc6283-kube-api-access-lgxsx\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:22:23.051602 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:23.051565 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2ttd\" (UniqueName: \"kubernetes.io/projected/cd1e671b-021c-428f-830c-9cf66ebb19c2-kube-api-access-x2ttd\") pod \"test-curl-7406-140152221477744\" (UID: \"cd1e671b-021c-428f-830c-9cf66ebb19c2\") " pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:23.062338 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:23.062296 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2ttd\" (UniqueName: \"kubernetes.io/projected/cd1e671b-021c-428f-830c-9cf66ebb19c2-kube-api-access-x2ttd\") pod \"test-curl-7406-140152221477744\" (UID: \"cd1e671b-021c-428f-830c-9cf66ebb19c2\") " pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:23.170051 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:23.170015 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:23.254767 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:23.254737 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df97d2dc-bd16-42aa-a67e-d2f750dc6283" path="/var/lib/kubelet/pods/df97d2dc-bd16-42aa-a67e-d2f750dc6283/volumes" Jul 10 18:22:23.308856 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:23.308815 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7406-140152221477744"] Jul 10 18:22:23.311165 ip-10-0-137-103 kubenswrapper[2556]: W0710 18:22:23.311138 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd1e671b_021c_428f_830c_9cf66ebb19c2.slice/crio-4457b2700be3cf2ea13e4b6cd438d69486af997ab989bc329d99cfe233cb5096 WatchSource:0}: Error finding container 4457b2700be3cf2ea13e4b6cd438d69486af997ab989bc329d99cfe233cb5096: Status 404 returned error can't find the container with id 4457b2700be3cf2ea13e4b6cd438d69486af997ab989bc329d99cfe233cb5096 Jul 10 18:22:23.541321 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:23.541238 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221476736" Jul 10 18:22:23.541321 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:23.541242 2556 scope.go:117] "RemoveContainer" containerID="cbc04b07103b16d2ac34279d36701db30206e37d9555487605e1df727e8d5d64" Jul 10 18:22:23.542664 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:23.542448 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7406-140152221477744" event={"ID":"cd1e671b-021c-428f-830c-9cf66ebb19c2","Type":"ContainerStarted","Data":"4457b2700be3cf2ea13e4b6cd438d69486af997ab989bc329d99cfe233cb5096"} Jul 10 18:22:24.123886 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.123822 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" podUID="09161c47-6950-485a-87df-417a3a68130e" containerName="main" containerID="cri-o://0b305e7d30c9fc26ddb9b85a270946e93547b9b4c2877df35f5ad69786ee4ab9" gracePeriod=45 Jul 10 18:22:24.390930 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.390902 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:22:24.549567 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.549530 2556 generic.go:356] "Generic (PLEG): container finished" podID="cd1e671b-021c-428f-830c-9cf66ebb19c2" containerID="912767a7901b8e7b4f417730c662623bb2c7ec7a4c9f61fe04ddb73ebafe7083" exitCode=0 Jul 10 18:22:24.549767 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.549616 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7406-140152221477744" event={"ID":"cd1e671b-021c-428f-830c-9cf66ebb19c2","Type":"ContainerDied","Data":"912767a7901b8e7b4f417730c662623bb2c7ec7a4c9f61fe04ddb73ebafe7083"} Jul 10 18:22:24.550793 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.550771 2556 generic.go:356] "Generic (PLEG): container finished" podID="09161c47-6950-485a-87df-417a3a68130e" containerID="0b305e7d30c9fc26ddb9b85a270946e93547b9b4c2877df35f5ad69786ee4ab9" exitCode=0 Jul 10 18:22:24.550897 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.550812 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" Jul 10 18:22:24.550897 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.550845 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" event={"ID":"09161c47-6950-485a-87df-417a3a68130e","Type":"ContainerDied","Data":"0b305e7d30c9fc26ddb9b85a270946e93547b9b4c2877df35f5ad69786ee4ab9"} Jul 10 18:22:24.550897 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.550870 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5" event={"ID":"09161c47-6950-485a-87df-417a3a68130e","Type":"ContainerDied","Data":"5d94efca7adde3df904d833a8f46c5a899a95719722ae7931942d92dbc462432"} Jul 10 18:22:24.550897 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.550887 2556 scope.go:117] "RemoveContainer" containerID="0b305e7d30c9fc26ddb9b85a270946e93547b9b4c2877df35f5ad69786ee4ab9" Jul 10 18:22:24.559179 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.559161 2556 scope.go:117] "RemoveContainer" containerID="0b305e7d30c9fc26ddb9b85a270946e93547b9b4c2877df35f5ad69786ee4ab9" Jul 10 18:22:24.559470 ip-10-0-137-103 kubenswrapper[2556]: E0710 18:22:24.559451 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b305e7d30c9fc26ddb9b85a270946e93547b9b4c2877df35f5ad69786ee4ab9\": container with ID starting with 0b305e7d30c9fc26ddb9b85a270946e93547b9b4c2877df35f5ad69786ee4ab9 not found: ID does not exist" containerID="0b305e7d30c9fc26ddb9b85a270946e93547b9b4c2877df35f5ad69786ee4ab9" Jul 10 18:22:24.559535 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.559476 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b305e7d30c9fc26ddb9b85a270946e93547b9b4c2877df35f5ad69786ee4ab9"} err="failed to get container status \"0b305e7d30c9fc26ddb9b85a270946e93547b9b4c2877df35f5ad69786ee4ab9\": rpc error: code = NotFound desc = could not find container \"0b305e7d30c9fc26ddb9b85a270946e93547b9b4c2877df35f5ad69786ee4ab9\": container with ID starting with 0b305e7d30c9fc26ddb9b85a270946e93547b9b4c2877df35f5ad69786ee4ab9 not found: ID does not exist" Jul 10 18:22:24.565416 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.565394 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv7p8\" (UniqueName: \"kubernetes.io/projected/09161c47-6950-485a-87df-417a3a68130e-kube-api-access-qv7p8\") pod \"09161c47-6950-485a-87df-417a3a68130e\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " Jul 10 18:22:24.565532 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.565452 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09161c47-6950-485a-87df-417a3a68130e-tls-certs\") pod \"09161c47-6950-485a-87df-417a3a68130e\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " Jul 10 18:22:24.565532 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.565499 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-model-cache\") pod \"09161c47-6950-485a-87df-417a3a68130e\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " Jul 10 18:22:24.565640 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.565534 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-dshm\") pod \"09161c47-6950-485a-87df-417a3a68130e\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " Jul 10 18:22:24.565640 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.565595 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-tmp-dir\") pod \"09161c47-6950-485a-87df-417a3a68130e\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " Jul 10 18:22:24.565640 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.565631 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-home\") pod \"09161c47-6950-485a-87df-417a3a68130e\" (UID: \"09161c47-6950-485a-87df-417a3a68130e\") " Jul 10 18:22:24.565845 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.565777 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-model-cache" (OuterVolumeSpecName: "model-cache") pod "09161c47-6950-485a-87df-417a3a68130e" (UID: "09161c47-6950-485a-87df-417a3a68130e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:24.565900 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.565887 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-model-cache\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:22:24.566067 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.566043 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09161c47-6950-485a-87df-417a3a68130e" (UID: "09161c47-6950-485a-87df-417a3a68130e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:24.566067 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.566055 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-home" (OuterVolumeSpecName: "home") pod "09161c47-6950-485a-87df-417a3a68130e" (UID: "09161c47-6950-485a-87df-417a3a68130e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:24.567732 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.567707 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09161c47-6950-485a-87df-417a3a68130e-kube-api-access-qv7p8" (OuterVolumeSpecName: "kube-api-access-qv7p8") pod "09161c47-6950-485a-87df-417a3a68130e" (UID: "09161c47-6950-485a-87df-417a3a68130e"). InnerVolumeSpecName "kube-api-access-qv7p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:22:24.567902 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.567874 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-dshm" (OuterVolumeSpecName: "dshm") pod "09161c47-6950-485a-87df-417a3a68130e" (UID: "09161c47-6950-485a-87df-417a3a68130e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:24.568139 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.568123 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09161c47-6950-485a-87df-417a3a68130e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09161c47-6950-485a-87df-417a3a68130e" (UID: "09161c47-6950-485a-87df-417a3a68130e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:22:24.667247 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.667141 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-tmp-dir\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:22:24.667247 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.667187 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-home\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:22:24.667247 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.667198 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv7p8\" (UniqueName: \"kubernetes.io/projected/09161c47-6950-485a-87df-417a3a68130e-kube-api-access-qv7p8\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:22:24.667247 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.667206 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09161c47-6950-485a-87df-417a3a68130e-tls-certs\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:22:24.667247 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.667215 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09161c47-6950-485a-87df-417a3a68130e-dshm\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:22:24.872009 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.871956 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5"] Jul 10 18:22:24.878200 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:24.878171 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/rate-test-model-b8bbaf-kserve-74b9b567dc-zvcm5"] Jul 10 18:22:25.255101 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.255061 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09161c47-6950-485a-87df-417a3a68130e" path="/var/lib/kubelet/pods/09161c47-6950-485a-87df-417a3a68130e/volumes" Jul 10 18:22:25.697904 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.697878 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:25.716379 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.716348 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7406-140152221477744_cd1e671b-021c-428f-830c-9cf66ebb19c2/test-curl-7406-140152221477744/0.log" Jul 10 18:22:25.736804 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.736777 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7406-140152221477744"] Jul 10 18:22:25.740641 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.740617 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7406-140152221477744"] Jul 10 18:22:25.858380 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.858347 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7406-140152221477744"] Jul 10 18:22:25.858698 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.858686 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09161c47-6950-485a-87df-417a3a68130e" containerName="main" Jul 10 18:22:25.858742 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.858699 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="09161c47-6950-485a-87df-417a3a68130e" containerName="main" Jul 10 18:22:25.858742 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.858716 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1e671b-021c-428f-830c-9cf66ebb19c2" containerName="test-curl-7406-140152221477744" Jul 10 18:22:25.858742 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.858724 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1e671b-021c-428f-830c-9cf66ebb19c2" containerName="test-curl-7406-140152221477744" Jul 10 18:22:25.858834 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.858785 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="09161c47-6950-485a-87df-417a3a68130e" containerName="main" Jul 10 18:22:25.858834 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.858796 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd1e671b-021c-428f-830c-9cf66ebb19c2" containerName="test-curl-7406-140152221477744" Jul 10 18:22:25.861707 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.861689 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:25.869871 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.869844 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7406-140152221477744"] Jul 10 18:22:25.879948 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.879923 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2ttd\" (UniqueName: \"kubernetes.io/projected/cd1e671b-021c-428f-830c-9cf66ebb19c2-kube-api-access-x2ttd\") pod \"cd1e671b-021c-428f-830c-9cf66ebb19c2\" (UID: \"cd1e671b-021c-428f-830c-9cf66ebb19c2\") " Jul 10 18:22:25.882148 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.882110 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1e671b-021c-428f-830c-9cf66ebb19c2-kube-api-access-x2ttd" (OuterVolumeSpecName: "kube-api-access-x2ttd") pod "cd1e671b-021c-428f-830c-9cf66ebb19c2" (UID: "cd1e671b-021c-428f-830c-9cf66ebb19c2"). InnerVolumeSpecName "kube-api-access-x2ttd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:22:25.980903 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.980860 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mft8n\" (UniqueName: \"kubernetes.io/projected/ae6d4913-1455-455e-88f2-c63504c5e529-kube-api-access-mft8n\") pod \"test-curl-7406-140152221477744\" (UID: \"ae6d4913-1455-455e-88f2-c63504c5e529\") " pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:25.981114 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:25.980926 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2ttd\" (UniqueName: \"kubernetes.io/projected/cd1e671b-021c-428f-830c-9cf66ebb19c2-kube-api-access-x2ttd\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:22:26.081550 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:26.081508 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mft8n\" (UniqueName: \"kubernetes.io/projected/ae6d4913-1455-455e-88f2-c63504c5e529-kube-api-access-mft8n\") pod \"test-curl-7406-140152221477744\" (UID: \"ae6d4913-1455-455e-88f2-c63504c5e529\") " pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:26.090180 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:26.090152 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mft8n\" (UniqueName: \"kubernetes.io/projected/ae6d4913-1455-455e-88f2-c63504c5e529-kube-api-access-mft8n\") pod \"test-curl-7406-140152221477744\" (UID: \"ae6d4913-1455-455e-88f2-c63504c5e529\") " pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:26.172051 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:26.171928 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:26.312749 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:26.312724 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7406-140152221477744"] Jul 10 18:22:26.314672 ip-10-0-137-103 kubenswrapper[2556]: W0710 18:22:26.314638 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae6d4913_1455_455e_88f2_c63504c5e529.slice/crio-f1165ea33782bd4dcbfd76fd4d369eff8fbfff52b56cfed78c51627e7c86fe68 WatchSource:0}: Error finding container f1165ea33782bd4dcbfd76fd4d369eff8fbfff52b56cfed78c51627e7c86fe68: Status 404 returned error can't find the container with id f1165ea33782bd4dcbfd76fd4d369eff8fbfff52b56cfed78c51627e7c86fe68 Jul 10 18:22:26.563241 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:26.563205 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7406-140152221477744" event={"ID":"ae6d4913-1455-455e-88f2-c63504c5e529","Type":"ContainerStarted","Data":"f1165ea33782bd4dcbfd76fd4d369eff8fbfff52b56cfed78c51627e7c86fe68"} Jul 10 18:22:26.564375 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:26.564349 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:26.564510 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:26.564380 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4457b2700be3cf2ea13e4b6cd438d69486af997ab989bc329d99cfe233cb5096" Jul 10 18:22:26.575664 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:26.575634 2556 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7406-140152221477744" oldPodUID="cd1e671b-021c-428f-830c-9cf66ebb19c2" podUID="ae6d4913-1455-455e-88f2-c63504c5e529" Jul 10 18:22:27.255284 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:27.255250 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd1e671b-021c-428f-830c-9cf66ebb19c2" path="/var/lib/kubelet/pods/cd1e671b-021c-428f-830c-9cf66ebb19c2/volumes" Jul 10 18:22:27.571189 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:27.571098 2556 generic.go:356] "Generic (PLEG): container finished" podID="ae6d4913-1455-455e-88f2-c63504c5e529" containerID="4a1613ea9c245cb08e171738ff565399ec8e772480e56ec0aada607104aa830b" exitCode=0 Jul 10 18:22:27.571189 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:27.571149 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7406-140152221477744" event={"ID":"ae6d4913-1455-455e-88f2-c63504c5e529","Type":"ContainerDied","Data":"4a1613ea9c245cb08e171738ff565399ec8e772480e56ec0aada607104aa830b"} Jul 10 18:22:28.723526 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:28.723500 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:28.742767 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:28.742742 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7406-140152221477744_ae6d4913-1455-455e-88f2-c63504c5e529/test-curl-7406-140152221477744/0.log" Jul 10 18:22:28.764020 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:28.763987 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7406-140152221477744"] Jul 10 18:22:28.769981 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:28.769940 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7406-140152221477744"] Jul 10 18:22:28.909328 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:28.909294 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mft8n\" (UniqueName: \"kubernetes.io/projected/ae6d4913-1455-455e-88f2-c63504c5e529-kube-api-access-mft8n\") pod \"ae6d4913-1455-455e-88f2-c63504c5e529\" (UID: \"ae6d4913-1455-455e-88f2-c63504c5e529\") " Jul 10 18:22:28.911559 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:28.911529 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6d4913-1455-455e-88f2-c63504c5e529-kube-api-access-mft8n" (OuterVolumeSpecName: "kube-api-access-mft8n") pod "ae6d4913-1455-455e-88f2-c63504c5e529" (UID: "ae6d4913-1455-455e-88f2-c63504c5e529"). InnerVolumeSpecName "kube-api-access-mft8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:22:29.010561 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:29.010527 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mft8n\" (UniqueName: \"kubernetes.io/projected/ae6d4913-1455-455e-88f2-c63504c5e529-kube-api-access-mft8n\") on node \"ip-10-0-137-103.ec2.internal\" DevicePath \"\"" Jul 10 18:22:29.255159 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:29.255067 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae6d4913-1455-455e-88f2-c63504c5e529" path="/var/lib/kubelet/pods/ae6d4913-1455-455e-88f2-c63504c5e529/volumes" Jul 10 18:22:29.582432 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:29.582348 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7406-140152221477744" Jul 10 18:22:29.582432 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:29.582365 2556 scope.go:117] "RemoveContainer" containerID="4a1613ea9c245cb08e171738ff565399ec8e772480e56ec0aada607104aa830b" Jul 10 18:22:33.257293 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:33.257200 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:22:43.258223 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:43.258192 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f55b4dd8-2qz9c_032ee506-19b6-43c8-9d49-f1cf24f19f4d/authorino/0.log" Jul 10 18:22:45.745751 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:45.745710 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-99mbl"] Jul 10 18:22:48.560389 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:48.560358 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dfdbc665d-n7dsm_250ddbcb-9cbd-469c-9df3-de593b7b736a/copy-manifests/0.log" Jul 10 18:22:48.567566 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:48.567540 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dfdbc665d-n7dsm_250ddbcb-9cbd-469c-9df3-de593b7b736a/manager/0.log" Jul 10 18:22:50.518148 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:50.518096 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f55b4dd8-2qz9c_032ee506-19b6-43c8-9d49-f1cf24f19f4d/authorino/0.log" Jul 10 18:22:50.854470 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:50.854441 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nzzqk_24516b57-3607-4080-b795-4ed5e576c8b4/kuadrant-console-plugin/0.log" Jul 10 18:22:51.207481 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:51.207391 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-99mbl_77995e24-aa95-4636-9117-e764fe6d390f/limitador/0.log" Jul 10 18:22:51.901558 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:51.901519 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b544bffd9-89d4w_04073a5d-860c-47f4-8f3d-ddfc2477fc62/kube-auth-proxy/0.log" Jul 10 18:22:53.148766 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:53.148730 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwd2zn_4ff4e515-56b8-470c-b9c5-f734b5c83279/main/0.log" Jul 10 18:22:53.363911 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:22:53.363883 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-ffvhw_151fabd2-9670-4a3c-b8c5-62a60e6c0ead/main/0.log" Jul 10 18:23:00.050525 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:00.050493 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wnspm_779679cf-3637-4afa-a580-767d44eb5e09/global-pull-secret-syncer/0.log" Jul 10 18:23:00.206739 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:00.206706 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v7gd5_6828147f-7c39-49c2-b67b-284254fe0357/konnectivity-agent/0.log" Jul 10 18:23:00.276709 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:00.276679 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-103.ec2.internal_5fecab31974d0fc256246ff6de1c24b9/haproxy/0.log" Jul 10 18:23:05.132946 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:05.132909 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f55b4dd8-2qz9c_032ee506-19b6-43c8-9d49-f1cf24f19f4d/authorino/0.log" Jul 10 18:23:05.231299 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:05.231260 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nzzqk_24516b57-3607-4080-b795-4ed5e576c8b4/kuadrant-console-plugin/0.log" Jul 10 18:23:05.480819 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:05.480730 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-99mbl_77995e24-aa95-4636-9117-e764fe6d390f/limitador/0.log" Jul 10 18:23:07.232248 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.232216 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905/alertmanager/0.log" Jul 10 18:23:07.261146 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.261113 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905/config-reloader/0.log" Jul 10 18:23:07.281755 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.281722 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905/kube-rbac-proxy-web/0.log" Jul 10 18:23:07.305468 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.305439 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905/kube-rbac-proxy/0.log" Jul 10 18:23:07.330014 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.329986 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905/kube-rbac-proxy-metric/0.log" Jul 10 18:23:07.352634 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.352608 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905/prom-label-proxy/0.log" Jul 10 18:23:07.373858 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.373832 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3f12f0d-1cf2-4cd5-8b9b-392a8bf28905/init-config-reloader/0.log" Jul 10 18:23:07.458397 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.458353 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-q8w4h_76091f5f-5621-4255-ac27-2aca61b38b53/cluster-monitoring-operator/0.log" Jul 10 18:23:07.486590 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.486507 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-wrdnb_6a5a8a6f-6afc-4114-998d-4102f3d4e91e/kube-state-metrics/0.log" Jul 10 18:23:07.507557 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.507531 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-wrdnb_6a5a8a6f-6afc-4114-998d-4102f3d4e91e/kube-rbac-proxy-main/0.log" Jul 10 18:23:07.540656 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.540632 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-wrdnb_6a5a8a6f-6afc-4114-998d-4102f3d4e91e/kube-rbac-proxy-self/0.log" Jul 10 18:23:07.611332 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.611305 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-9q2mt_2aa5cef4-904a-489d-8683-41a01dc7030d/monitoring-plugin/0.log" Jul 10 18:23:07.806134 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.806059 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w8nqr_61455718-aab9-4990-b93b-2c1d7be1532d/node-exporter/0.log" Jul 10 18:23:07.824204 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.824181 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w8nqr_61455718-aab9-4990-b93b-2c1d7be1532d/kube-rbac-proxy/0.log" Jul 10 18:23:07.845630 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.845606 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w8nqr_61455718-aab9-4990-b93b-2c1d7be1532d/init-textfile/0.log" Jul 10 18:23:07.887958 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.887925 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-gt6gl_f1b70818-5e06-41c7-88fc-a321618a3349/kube-rbac-proxy-main/0.log" Jul 10 18:23:07.908814 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.908785 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-gt6gl_f1b70818-5e06-41c7-88fc-a321618a3349/kube-rbac-proxy-self/0.log" Jul 10 18:23:07.932998 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:07.932954 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-gt6gl_f1b70818-5e06-41c7-88fc-a321618a3349/openshift-state-metrics/0.log" Jul 10 18:23:10.163180 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.163150 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/1.log" Jul 10 18:23:10.171459 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.171426 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xcxtg_672f8194-f0a2-460c-891e-1c6e7ca21f81/console-operator/2.log" Jul 10 18:23:10.532761 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.532681 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4"] Jul 10 18:23:10.533090 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.533064 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6d4913-1455-455e-88f2-c63504c5e529" containerName="test-curl-7406-140152221477744" Jul 10 18:23:10.533090 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.533084 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6d4913-1455-455e-88f2-c63504c5e529" containerName="test-curl-7406-140152221477744" Jul 10 18:23:10.533290 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.533174 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6d4913-1455-455e-88f2-c63504c5e529" containerName="test-curl-7406-140152221477744" Jul 10 18:23:10.536485 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.536464 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.539152 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.539108 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kb8lk\"/\"openshift-service-ca.crt\"" Jul 10 18:23:10.540155 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.540138 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kb8lk\"/\"default-dockercfg-264mv\"" Jul 10 18:23:10.540155 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.540149 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kb8lk\"/\"kube-root-ca.crt\"" Jul 10 18:23:10.544941 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.544916 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4"] Jul 10 18:23:10.692096 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.692059 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2l5v\" (UniqueName: \"kubernetes.io/projected/cd54ed2d-d200-4ef2-9e32-6883492f30af-kube-api-access-n2l5v\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.692282 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.692106 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cd54ed2d-d200-4ef2-9e32-6883492f30af-proc\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.692282 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.692187 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd54ed2d-d200-4ef2-9e32-6883492f30af-lib-modules\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.692282 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.692234 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cd54ed2d-d200-4ef2-9e32-6883492f30af-podres\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.692393 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.692296 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd54ed2d-d200-4ef2-9e32-6883492f30af-sys\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.752864 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.752838 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7b557c49f5-8qnm2_0c0a9e34-d2f8-44f7-b8ef-a6c1d37c71e2/console/0.log" Jul 10 18:23:10.792846 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.792726 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd54ed2d-d200-4ef2-9e32-6883492f30af-sys\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.792846 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.792776 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2l5v\" (UniqueName: \"kubernetes.io/projected/cd54ed2d-d200-4ef2-9e32-6883492f30af-kube-api-access-n2l5v\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.792846 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.792814 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cd54ed2d-d200-4ef2-9e32-6883492f30af-proc\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.793280 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.792854 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd54ed2d-d200-4ef2-9e32-6883492f30af-sys\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.793280 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.792859 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd54ed2d-d200-4ef2-9e32-6883492f30af-lib-modules\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.793280 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.792916 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cd54ed2d-d200-4ef2-9e32-6883492f30af-proc\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.793280 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.792950 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cd54ed2d-d200-4ef2-9e32-6883492f30af-podres\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.793280 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.793001 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd54ed2d-d200-4ef2-9e32-6883492f30af-lib-modules\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.793280 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.793100 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cd54ed2d-d200-4ef2-9e32-6883492f30af-podres\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.803789 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.803766 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2l5v\" (UniqueName: \"kubernetes.io/projected/cd54ed2d-d200-4ef2-9e32-6883492f30af-kube-api-access-n2l5v\") pod \"perf-node-gather-daemonset-958k4\" (UID: \"cd54ed2d-d200-4ef2-9e32-6883492f30af\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.847531 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.847496 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:10.982485 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:10.982460 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4"] Jul 10 18:23:10.984841 ip-10-0-137-103 kubenswrapper[2556]: W0710 18:23:10.984802 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcd54ed2d_d200_4ef2_9e32_6883492f30af.slice/crio-cbacedc2268befd7b87e3fe4ce82349348b17a437123a774961530c2abc02d6a WatchSource:0}: Error finding container cbacedc2268befd7b87e3fe4ce82349348b17a437123a774961530c2abc02d6a: Status 404 returned error can't find the container with id cbacedc2268befd7b87e3fe4ce82349348b17a437123a774961530c2abc02d6a Jul 10 18:23:11.349893 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:11.349803 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-9xdxz_1539cdfd-1ef2-4cbb-86ef-9eaf178c19e3/volume-data-source-validator/0.log" Jul 10 18:23:11.801111 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:11.801076 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" event={"ID":"cd54ed2d-d200-4ef2-9e32-6883492f30af","Type":"ContainerStarted","Data":"2c91459583e0ca690989eec8d587a5369b365cf60f5b0d57e6f1286167c5b28f"} Jul 10 18:23:11.801293 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:11.801132 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" event={"ID":"cd54ed2d-d200-4ef2-9e32-6883492f30af","Type":"ContainerStarted","Data":"cbacedc2268befd7b87e3fe4ce82349348b17a437123a774961530c2abc02d6a"} Jul 10 18:23:11.801293 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:11.801223 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:11.817622 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:11.817566 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" podStartSLOduration=1.817550532 podStartE2EDuration="1.817550532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:23:11.815336795 +0000 UTC m=+2641.089663516" watchObservedRunningTime="2026-07-10 18:23:11.817550532 +0000 UTC m=+2641.091877226" Jul 10 18:23:12.383102 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:12.383063 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8dblb_1f6a4ab1-f10d-4111-812a-90d3f35f3484/dns/0.log" Jul 10 18:23:12.407271 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:12.407246 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8dblb_1f6a4ab1-f10d-4111-812a-90d3f35f3484/kube-rbac-proxy/0.log" Jul 10 18:23:12.496917 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:12.496880 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-flzpz_3a04a38b-bb00-4ab0-b1d2-433b32e982e2/dns-node-resolver/0.log" Jul 10 18:23:13.075107 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:13.075077 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4sdqt_67a35b3c-e9c2-45f7-9518-9754c6015eb0/node-ca/0.log" Jul 10 18:23:14.133274 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:14.133248 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b544bffd9-89d4w_04073a5d-860c-47f4-8f3d-ddfc2477fc62/kube-auth-proxy/0.log" Jul 10 18:23:15.022582 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:15.022552 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x7gfp_9035ea2d-129f-462f-8581-bfaf86b87718/serve-healthcheck-canary/0.log" Jul 10 18:23:15.640776 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:15.640747 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hs657_3d1f472c-aae2-436c-944e-754bafcadbdf/kube-rbac-proxy/0.log" Jul 10 18:23:15.662373 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:15.662333 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hs657_3d1f472c-aae2-436c-944e-754bafcadbdf/exporter/0.log" Jul 10 18:23:15.683749 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:15.683721 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hs657_3d1f472c-aae2-436c-944e-754bafcadbdf/extractor/0.log" Jul 10 18:23:17.817182 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:17.817150 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-958k4" Jul 10 18:23:18.104857 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:18.104831 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dfdbc665d-n7dsm_250ddbcb-9cbd-469c-9df3-de593b7b736a/manager/0.log" Jul 10 18:23:18.122420 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:18.122396 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dfdbc665d-n7dsm_250ddbcb-9cbd-469c-9df3-de593b7b736a/copy-manifests/0.log" Jul 10 18:23:19.555161 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:19.555129 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7659478ddc-zzqfx_6199be07-663c-4792-afaa-ca98a12c9902/manager/0.log" Jul 10 18:23:19.586105 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:19.586070 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-qklzj_f131c16c-eaee-4ace-8c55-a2d8e976fb9c/openshift-lws-operator/0.log" Jul 10 18:23:20.873625 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:20.873597 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b544bffd9-89d4w_04073a5d-860c-47f4-8f3d-ddfc2477fc62/kube-auth-proxy/0.log" Jul 10 18:23:22.312213 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:22.312177 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b544bffd9-89d4w_04073a5d-860c-47f4-8f3d-ddfc2477fc62/kube-auth-proxy/0.log" Jul 10 18:23:24.879830 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:24.879797 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-pdjp8_fa889ec6-c374-4a61-9465-061926984d83/migrator/0.log" Jul 10 18:23:24.901153 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:24.901127 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-pdjp8_fa889ec6-c374-4a61-9465-061926984d83/graceful-termination/0.log" Jul 10 18:23:26.497044 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:26.497011 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ntwdg_63f4fcc3-2ddc-40fa-8a79-7f02580ce489/kube-multus-additional-cni-plugins/0.log" Jul 10 18:23:26.517452 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:26.517429 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ntwdg_63f4fcc3-2ddc-40fa-8a79-7f02580ce489/egress-router-binary-copy/0.log" Jul 10 18:23:26.537747 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:26.537724 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ntwdg_63f4fcc3-2ddc-40fa-8a79-7f02580ce489/cni-plugins/0.log" Jul 10 18:23:26.559155 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:26.559100 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ntwdg_63f4fcc3-2ddc-40fa-8a79-7f02580ce489/bond-cni-plugin/0.log" Jul 10 18:23:26.577455 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:26.577427 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ntwdg_63f4fcc3-2ddc-40fa-8a79-7f02580ce489/routeoverride-cni/0.log" Jul 10 18:23:26.596336 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:26.596254 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ntwdg_63f4fcc3-2ddc-40fa-8a79-7f02580ce489/whereabouts-cni-bincopy/0.log" Jul 10 18:23:26.614465 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:26.614437 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ntwdg_63f4fcc3-2ddc-40fa-8a79-7f02580ce489/whereabouts-cni/0.log" Jul 10 18:23:26.711149 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:26.711119 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ln7km_0ca99128-dbcc-4b37-98d3-4d9d70934cc8/kube-multus/0.log" Jul 10 18:23:26.771277 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:26.771192 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p58j7_17b57439-069e-42d9-b59f-976e14670388/network-metrics-daemon/0.log" Jul 10 18:23:26.787242 ip-10-0-137-103 kubenswrapper[2556]: I0710 18:23:26.787211 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p58j7_17b57439-069e-42d9-b59f-976e14670388/kube-rbac-proxy/0.log"