Jul 06 20:35:07.726394 ip-10-0-132-244 systemd[1]: Starting Kubernetes Kubelet... Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.155146 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161585 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161596 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161599 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161602 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161606 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161608 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161612 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:35:08.206663 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161615 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161618 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161621 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161624 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161627 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161629 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161632 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161636 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161639 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161641 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161644 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161647 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161649 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161652 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161654 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161656 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161659 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161662 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161666 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161670 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:35:08.491500 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161673 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:35:08.233265 ip-10-0-132-244 systemd[1]: Started Kubernetes Kubelet. Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161676 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161678 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161681 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161683 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161686 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161688 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161691 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161693 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161697 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161700 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161702 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161705 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161708 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161711 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161714 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161716 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161719 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161723 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161728 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:35:08.498197 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161731 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161734 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161737 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161739 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161742 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161744 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161747 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161749 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161752 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161754 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161757 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161760 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161763 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161766 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161769 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161771 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161774 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161777 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161779 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:35:08.499438 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161782 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161784 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161787 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161792 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161795 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161798 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161801 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161804 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161807 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161810 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161812 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161815 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161817 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161820 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161823 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161825 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161828 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161831 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161833 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161836 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:35:08.500475 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161838 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161841 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161843 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161846 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161848 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161851 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161853 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161856 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161859 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161862 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161864 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161867 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.161869 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162579 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162586 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162589 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162592 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162596 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162599 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162602 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:35:08.501334 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162605 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162607 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162610 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162613 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162615 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162618 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162620 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162623 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162625 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162628 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162630 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162633 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162635 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162638 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162641 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162643 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162646 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162648 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162651 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162653 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:35:08.502389 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162656 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162658 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162661 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162663 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162666 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162668 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162671 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162673 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162676 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162678 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162681 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162690 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162693 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162696 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162698 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162708 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162711 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162713 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162716 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:35:08.503448 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162719 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162721 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162724 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162726 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162729 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162731 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162734 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162737 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162739 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162742 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162744 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162746 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162750 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162754 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162756 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162759 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162762 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162765 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162767 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:35:08.504489 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162770 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162773 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162775 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162777 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162780 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162782 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162790 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162793 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162795 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162798 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162800 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162803 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162805 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162808 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162810 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162812 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162814 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162818 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162820 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162824 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:35:08.505339 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162826 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162828 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162831 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162833 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162836 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162839 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162842 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162847 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162849 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162852 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162856 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162859 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162862 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.162865 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163359 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163365 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163370 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163372 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163381 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163384 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:35:08.506294 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163386 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163388 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163391 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163394 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163397 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163399 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163402 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163404 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163407 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163409 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163413 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163416 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163419 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163422 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163424 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163427 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163429 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163433 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:35:08.507145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163436 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163441 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163444 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163447 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163450 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163453 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163455 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163458 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163460 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163463 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163467 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163469 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163472 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163479 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163482 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163484 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163487 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163489 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163491 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163494 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:35:08.507995 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163496 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163499 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163501 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163504 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163506 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163508 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163510 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163513 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163515 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163518 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163520 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163522 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163525 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163528 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163531 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163533 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163536 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163538 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163540 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163543 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:35:08.509045 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163545 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163547 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163551 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163553 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163556 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163558 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163569 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163572 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163574 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163577 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163579 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163581 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163584 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163586 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163589 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163591 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163593 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163596 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163598 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163601 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:35:08.510113 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163603 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163606 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163608 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163610 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163613 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163617 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163619 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163621 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163624 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163627 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163629 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163631 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163633 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163636 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.163639 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163705 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163716 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163728 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163732 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163742 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163745 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 20:35:08.510891 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163749 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163753 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163756 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163760 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163763 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163766 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163769 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163771 2579 flags.go:64] FLAG: --cgroup-root="" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163774 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163777 2579 flags.go:64] FLAG: --client-ca-file="" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163780 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163783 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163789 2579 flags.go:64] FLAG: --cluster-domain="" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163791 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163794 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163797 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163802 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163806 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163808 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163811 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163814 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163817 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163820 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163822 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 20:35:08.511944 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163826 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163830 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163834 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163837 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163839 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163842 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163845 2579 flags.go:64] FLAG: --enable-server="true" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163855 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163861 2579 flags.go:64] FLAG: --event-burst="100" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163864 2579 flags.go:64] FLAG: --event-qps="50" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163867 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163870 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163873 2579 flags.go:64] FLAG: --eviction-hard="" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163876 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163878 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163881 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163884 2579 flags.go:64] FLAG: --eviction-soft="" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163887 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163890 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163892 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163895 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163898 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163900 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163903 2579 flags.go:64] FLAG: --feature-gates="" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163908 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 20:35:08.512991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163911 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163914 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163917 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163920 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163922 2579 flags.go:64] FLAG: --help="false" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163925 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163928 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163931 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163935 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163938 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163941 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163944 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163950 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163953 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163956 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163964 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163967 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163970 2579 flags.go:64] FLAG: --kube-reserved="" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163973 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163976 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163979 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163981 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163984 2579 flags.go:64] FLAG: --lock-file="" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163987 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 20:35:08.514030 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163989 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163992 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.163997 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164000 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164002 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164005 2579 flags.go:64] FLAG: --logging-format="text" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164007 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164012 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164014 2579 flags.go:64] FLAG: --manifest-url="" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164017 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164021 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164023 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164027 2579 flags.go:64] FLAG: --max-pods="110" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164030 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164033 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164035 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164039 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164042 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164045 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164048 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164054 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164057 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164060 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 20:35:08.515118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164063 2579 flags.go:64] FLAG: --pod-cidr="" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164071 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164077 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164080 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164082 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164085 2579 flags.go:64] FLAG: --port="10250" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164088 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164091 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cc338e9a9a1d9f21" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164094 2579 flags.go:64] FLAG: --qos-reserved="" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164097 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164113 2579 flags.go:64] FLAG: --register-node="true" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164117 2579 flags.go:64] FLAG: --register-with-taints="" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164120 2579 flags.go:64] FLAG: --registry-burst="10" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164124 2579 flags.go:64] FLAG: --registry-qps="5" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164128 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164132 2579 flags.go:64] FLAG: --reserved-memory="" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164137 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164142 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164144 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164147 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164151 2579 flags.go:64] FLAG: --runonce="false" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164154 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164157 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164160 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164162 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 20:35:08.516377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164166 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164169 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164172 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164175 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164178 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164181 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164183 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164186 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164189 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164198 2579 flags.go:64] FLAG: --system-cgroups="" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164201 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164206 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164208 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164211 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164217 2579 flags.go:64] FLAG: --tls-min-version="" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164220 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164222 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164225 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164228 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164231 2579 flags.go:64] FLAG: --v="2" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164234 2579 flags.go:64] FLAG: --version="false" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164238 2579 flags.go:64] FLAG: --vmodule="" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164241 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.164244 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164399 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:35:08.519525 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164402 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164405 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164408 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164411 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164413 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164417 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164420 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164424 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164427 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164430 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164432 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164435 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164437 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164440 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164442 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164444 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164447 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164455 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164458 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:35:08.520580 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164460 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164463 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164465 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164468 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164470 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164473 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164476 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164479 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164482 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164484 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164487 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164489 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164493 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164496 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164498 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164501 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164503 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164506 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164508 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:35:08.524069 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164511 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164514 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164517 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164519 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164521 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164524 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164527 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164529 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164532 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164534 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164543 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164546 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164553 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164556 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164558 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164561 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164563 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164566 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164568 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164570 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:35:08.524617 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164573 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164575 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164578 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164580 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164582 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164587 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164589 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164592 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164594 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164596 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164599 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164601 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164604 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164608 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164610 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164612 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164614 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164617 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164619 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164621 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:35:08.525139 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164624 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164626 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164628 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164631 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164633 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164641 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164645 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164648 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164651 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164653 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164656 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164659 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164662 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164664 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164667 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164669 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164671 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164675 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164677 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:35:08.709125 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.164680 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.165394 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.175078 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.175096 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.175134 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.175140 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175197 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175203 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175206 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175209 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175212 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175215 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175218 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175221 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175224 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:35:08.710081 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175228 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175232 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175235 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175239 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175242 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175244 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175247 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175250 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175253 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175257 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175260 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175263 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175265 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175268 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175271 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175274 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175276 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175279 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175282 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175284 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:35:08.710929 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175287 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175290 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175292 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175295 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175298 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175300 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175302 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175305 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175307 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175310 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175312 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175315 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175318 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175321 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175323 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175328 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175330 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175333 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175335 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175338 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:35:08.711762 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175341 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175347 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175350 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175353 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175356 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175359 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175362 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175364 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175367 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175369 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175372 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175374 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175377 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175379 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175382 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175385 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175388 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175390 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175392 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:35:08.712597 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175395 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175398 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175400 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175403 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175405 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175408 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175410 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175413 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175417 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175420 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175422 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175427 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175431 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175434 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175438 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175441 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175444 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175447 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175450 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:35:08.713446 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175452 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175455 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175458 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175461 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175464 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175466 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175469 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175472 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175474 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175477 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175480 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175482 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.175488 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175614 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175619 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:35:08.714299 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175622 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175625 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175628 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175631 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175634 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175636 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175641 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175643 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175646 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175649 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175652 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175655 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175657 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175661 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175663 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175666 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175668 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175671 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175673 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175676 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:35:08.714926 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175678 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175681 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175684 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175687 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175689 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175692 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175694 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175697 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175699 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175702 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175704 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175707 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175709 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175712 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175714 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175717 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175720 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175723 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175728 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:35:08.715977 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175730 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175733 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175735 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175738 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175740 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175743 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175747 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175749 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175752 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175754 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175757 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175759 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175762 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175764 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175767 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175770 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175772 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175775 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175777 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175779 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:35:08.716552 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175782 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175785 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175787 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175790 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175792 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175794 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175797 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175799 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175802 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175804 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175807 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175811 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175814 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175816 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175819 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175821 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175825 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175829 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175833 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:35:08.717154 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175836 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175838 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175842 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175845 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175847 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175850 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175853 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175855 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175858 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175860 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175863 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175865 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175868 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175870 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175873 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175875 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175891 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175895 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:35:08.717644 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:08.175898 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:35:08.718153 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.175903 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:35:08.718153 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.176003 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 20:35:08.718153 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.180953 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 20:35:08.718153 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.182732 2579 server.go:1019] "Starting client certificate rotation" Jul 06 20:35:08.718153 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.182831 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 20:35:08.718153 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.182857 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 20:35:08.718153 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.186574 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 20:35:08.718153 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.187764 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 20:35:08.718153 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.193751 2579 log.go:25] "Validated CRI v1 runtime API" Jul 06 20:35:08.718153 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.197899 2579 log.go:25] "Validated CRI v1 image API" Jul 06 20:35:08.718153 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.199990 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 20:35:08.718153 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.203843 2579 fs.go:135] Filesystem UUIDs: map[05e86f81-b99c-4c57-b0dc-0604414cda36:/dev/nvme0n1p4 1e88fe79-2f2b-4c7b-92a0-9a795e98d65f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 06 20:35:08.718500 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.203856 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 20:35:08.718637 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.209313 2579 manager.go:216] Machine: {Timestamp:2026-07-06 20:35:08.207793008 +0000 UTC m=+0.370007966 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3204285 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d78992cfd061c07918abf38b01df8 SystemUUID:ec2d7899-2cfd-061c-0791-8abf38b01df8 BootID:ae51f2da-468a-4472-8d6d-7d42178d437e Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5f:81:5a:ae:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5f:81:5a:ae:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:79:d5:2b:79:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 20:35:08.718637 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.209407 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 20:35:08.718637 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.209483 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 20:35:08.718637 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.213419 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 20:35:08.718637 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.213444 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-244.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 20:35:08.718637 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.214229 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 20:35:08.718637 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.214238 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 20:35:08.718637 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.214246 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.214954 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.216497 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.218125 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.219891 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.219995 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.220026 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.220035 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.220048 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.220056 2579 kubelet.go:398] "Adding apiserver pod source" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.220064 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.220582 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.220610 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.224177 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.226591 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.226617 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228029 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228053 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228061 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228066 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228074 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228079 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228086 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228095 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228119 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228125 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228133 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228138 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228980 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.228988 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.231850 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-244.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.232006 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.232032 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.232445 2579 server.go:1268] "Started kubelet" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.232521 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.232561 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.232570 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.234594 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.234992 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 20:35:08.718901 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.238603 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.238653 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-244.ec2.internal.18bfcdaa2c3bfd76 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-244.ec2.internal,UID:ip-10-0-132-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-244.ec2.internal,},FirstTimestamp:2026-07-06 20:35:08.232269174 +0000 UTC m=+0.394484125,LastTimestamp:2026-07-06 20:35:08.232269174 +0000 UTC m=+0.394484125,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-244.ec2.internal,}" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.243538 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.243546 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.243666 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.244211 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.244209 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.244248 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.244326 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.244335 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.244415 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.244428 2579 factory.go:55] Registering systemd factory Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.244434 2579 factory.go:223] Registration of the systemd container factory successfully Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.244626 2579 factory.go:153] Registering CRI-O factory Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.244637 2579 factory.go:223] Registration of the crio container factory successfully Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.244655 2579 factory.go:103] Registering Raw factory Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.244667 2579 manager.go:1196] Started watching for new ooms in manager Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.245155 2579 manager.go:349] Starting recovery of all containers Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.249862 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.249921 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.260047 2579 manager.go:354] Recovery completed Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.264534 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.266503 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.266546 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.266561 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.267028 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.267038 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.267052 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.267075 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.267717 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.267744 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.267755 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.268205 2579 policy_none.go:49] "None policy: Start" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.268222 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 20:35:08.719910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.268234 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.268932 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-244.ec2.internal.18bfcdaa2e46bfaf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-244.ec2.internal,UID:ip-10-0-132-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-244.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-244.ec2.internal,},FirstTimestamp:2026-07-06 20:35:08.266528687 +0000 UTC m=+0.428743642,LastTimestamp:2026-07-06 20:35:08.266528687 +0000 UTC m=+0.428743642,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-244.ec2.internal,}" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.269247 2579 policy_none.go:47] "Start" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.274798 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ft7nw" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.275986 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-244.ec2.internal.18bfcdaa2e471e26 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-244.ec2.internal,UID:ip-10-0-132-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-244.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-244.ec2.internal,},FirstTimestamp:2026-07-06 20:35:08.26655287 +0000 UTC m=+0.428767823,LastTimestamp:2026-07-06 20:35:08.26655287 +0000 UTC m=+0.428767823,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-244.ec2.internal,}" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.278600 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-244.ec2.internal.18bfcdaa2e475390 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-244.ec2.internal,UID:ip-10-0-132-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-244.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-244.ec2.internal,},FirstTimestamp:2026-07-06 20:35:08.266566544 +0000 UTC m=+0.428781498,LastTimestamp:2026-07-06 20:35:08.266566544 +0000 UTC m=+0.428781498,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-244.ec2.internal,}" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.283693 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ft7nw" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.339625 2579 manager.go:337] "Starting Device Plugin manager" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.339655 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.339664 2579 server.go:87] "Starting device plugin registration server" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.339920 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.339933 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.340032 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.340128 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.340139 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.340547 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.340598 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-244.ec2.internal\" not found" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.440756 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.441674 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:08.721216 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.441698 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.441707 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.441731 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.451556 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.451573 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-244.ec2.internal\": node \"ip-10-0-132-244.ec2.internal\" not found" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.488930 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.545131 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.546435 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.546478 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.546503 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.546541 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.549083 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.589465 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.647043 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal"] Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.647776 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.649126 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.649149 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.649162 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.650285 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.650441 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.650467 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.651007 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.651023 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.651035 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.651385 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.651404 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.651414 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.652092 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.652129 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.652723 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.652743 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.652753 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.679652 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-244.ec2.internal\" not found" node="ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.721776 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.684164 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-244.ec2.internal\" not found" node="ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.901648 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.690305 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 06 20:35:08.901648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.746360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9517028d6d269b9d2b51313aaffc1f4e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-244.ec2.internal\" (UID: \"9517028d6d269b9d2b51313aaffc1f4e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.901648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.746385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/75d37fda4a913c36c0e6c4c580018d46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal\" (UID: \"75d37fda4a913c36c0e6c4c580018d46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.901648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.746403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d37fda4a913c36c0e6c4c580018d46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal\" (UID: \"75d37fda4a913c36c0e6c4c580018d46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.901648 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.791295 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 06 20:35:08.901648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.849938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9517028d6d269b9d2b51313aaffc1f4e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-244.ec2.internal\" (UID: \"9517028d6d269b9d2b51313aaffc1f4e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.901648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.849967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/75d37fda4a913c36c0e6c4c580018d46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal\" (UID: \"75d37fda4a913c36c0e6c4c580018d46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.901648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.849983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d37fda4a913c36c0e6c4c580018d46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal\" (UID: \"75d37fda4a913c36c0e6c4c580018d46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.901648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.850029 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9517028d6d269b9d2b51313aaffc1f4e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-244.ec2.internal\" (UID: \"9517028d6d269b9d2b51313aaffc1f4e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.901648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.850068 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d37fda4a913c36c0e6c4c580018d46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal\" (UID: \"75d37fda4a913c36c0e6c4c580018d46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.901648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.850087 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/75d37fda4a913c36c0e6c4c580018d46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal\" (UID: \"75d37fda4a913c36c0e6c4c580018d46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 06 20:35:08.901648 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.892050 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 06 20:35:09.022127 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.984915 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" Jul 06 20:35:09.022127 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:08.987048 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 06 20:35:09.022127 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:08.992423 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 06 20:35:09.199680 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.047529 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:35:09.199680 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.144385 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" Jul 06 20:35:09.199680 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.158340 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 20:35:09.199680 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.160047 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 06 20:35:09.199680 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.161870 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:35:09.199680 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.176163 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 20:35:09.199680 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.182144 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 20:35:09.199680 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.182256 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 20:35:09.199680 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.182287 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.220801 2579 apiserver.go:52] "Watching apiserver" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.231982 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.233116 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kfgw5","kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5","openshift-cluster-node-tuning-operator/tuned-zwvpx","openshift-dns/node-resolver-nsdj2","openshift-image-registry/node-ca-whgpf","openshift-multus/multus-additional-cni-plugins-9frbs","openshift-multus/multus-g5469","kube-system/konnectivity-agent-pqs8k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal","openshift-multus/network-metrics-daemon-c5bs6","openshift-network-diagnostics/network-check-target-m6hmf","openshift-network-operator/iptables-alerter-6wgrb","openshift-ovn-kubernetes/ovnkube-node-kqkjl"] Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.234457 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.235409 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6wgrb" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.236344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.236739 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.237294 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.238184 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nsdj2" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.238229 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.238262 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.238339 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.238539 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.238603 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.238672 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bnd65\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.238694 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rz2j7\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.238713 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.238408 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.238854 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.238996 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qjz2g\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.239050 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.239096 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.239301 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-whgpf" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.240475 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.240924 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.243499 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.243660 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.244134 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 20:35:10.389708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.244159 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pqs8k" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.244455 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.244463 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.244136 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5nls2\"" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.244508 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:09.244524 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.244715 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.245760 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g5469" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.246747 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:09.246813 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.247299 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-25dbw\"" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.247453 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-spzvm\"" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.247726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:09.247768 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.249456 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.249580 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.249709 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zs6kf\"" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251234 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljv69\" (UniqueName: \"kubernetes.io/projected/157e3ec8-eb18-430f-b4b5-397ab0281e2a-kube-api-access-ljv69\") pod \"node-resolver-nsdj2\" (UID: \"157e3ec8-eb18-430f-b4b5-397ab0281e2a\") " pod="openshift-dns/node-resolver-nsdj2" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a989d7ea-35b8-47ea-8942-a5af5d9718ab-cnibin\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-sys\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/157e3ec8-eb18-430f-b4b5-397ab0281e2a-hosts-file\") pod \"node-resolver-nsdj2\" (UID: \"157e3ec8-eb18-430f-b4b5-397ab0281e2a\") " pod="openshift-dns/node-resolver-nsdj2" Jul 06 20:35:10.391076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-registration-dir\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251338 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-slash\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-run-netns\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a989d7ea-35b8-47ea-8942-a5af5d9718ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-kubernetes\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-sysctl-conf\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251461 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-run\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f0d0297-cff0-4b11-b42c-af7c14b392a4-serviceca\") pod \"node-ca-whgpf\" (UID: \"6f0d0297-cff0-4b11-b42c-af7c14b392a4\") " pod="openshift-image-registry/node-ca-whgpf" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-kubelet\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251534 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-sysctl-d\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-systemd\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-lib-modules\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.391743 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251601 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-run-ovn\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.392230 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-device-dir\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.392230 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-node-log\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.392230 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ba8e8ab-bd92-4f3e-9266-c41227aadedb-host-slash\") pod \"iptables-alerter-6wgrb\" (UID: \"9ba8e8ab-bd92-4f3e-9266-c41227aadedb\") " pod="openshift-network-operator/iptables-alerter-6wgrb" Jul 06 20:35:10.392230 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-var-lib-kubelet\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.392230 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251687 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-host\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.392230 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81dde24b-8630-4c06-8ce9-542aa3db5276-tmp\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.392230 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251756 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf54w\" (UniqueName: \"kubernetes.io/projected/81dde24b-8630-4c06-8ce9-542aa3db5276-kube-api-access-qf54w\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.392230 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-run-systemd\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.392230 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a989d7ea-35b8-47ea-8942-a5af5d9718ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.392230 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-modprobe-d\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.392230 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-etc-openvswitch\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.392230 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-cni-netd\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.392230 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.392682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/366ffea7-6944-47a4-9d15-bcf3aca0756c-env-overrides\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.392682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qrzz\" (UniqueName: \"kubernetes.io/projected/366ffea7-6944-47a4-9d15-bcf3aca0756c-kube-api-access-4qrzz\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.392682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bcc5ed5-4a67-416b-bebf-93316f005012-konnectivity-ca\") pod \"konnectivity-agent-pqs8k\" (UID: \"8bcc5ed5-4a67-416b-bebf-93316f005012\") " pod="kube-system/konnectivity-agent-pqs8k" Jul 06 20:35:10.392682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.251991 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77j94\" (UniqueName: \"kubernetes.io/projected/6f0d0297-cff0-4b11-b42c-af7c14b392a4-kube-api-access-77j94\") pod \"node-ca-whgpf\" (UID: \"6f0d0297-cff0-4b11-b42c-af7c14b392a4\") " pod="openshift-image-registry/node-ca-whgpf" Jul 06 20:35:10.392682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2fb9f72c-98b1-4441-9c7d-57263e220a21-dbus\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:10.392682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252037 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcwp9\" (UniqueName: \"kubernetes.io/projected/9ba8e8ab-bd92-4f3e-9266-c41227aadedb-kube-api-access-rcwp9\") pod \"iptables-alerter-6wgrb\" (UID: \"9ba8e8ab-bd92-4f3e-9266-c41227aadedb\") " pod="openshift-network-operator/iptables-alerter-6wgrb" Jul 06 20:35:10.392682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/366ffea7-6944-47a4-9d15-bcf3aca0756c-ovn-node-metrics-cert\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.392682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:10.392682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a989d7ea-35b8-47ea-8942-a5af5d9718ab-system-cni-dir\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.392682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252142 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ba8e8ab-bd92-4f3e-9266-c41227aadedb-iptables-alerter-script\") pod \"iptables-alerter-6wgrb\" (UID: \"9ba8e8ab-bd92-4f3e-9266-c41227aadedb\") " pod="openshift-network-operator/iptables-alerter-6wgrb" Jul 06 20:35:10.392682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-systemd-units\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.392682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-cni-bin\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.392682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bcc5ed5-4a67-416b-bebf-93316f005012-agent-certs\") pod \"konnectivity-agent-pqs8k\" (UID: \"8bcc5ed5-4a67-416b-bebf-93316f005012\") " pod="kube-system/konnectivity-agent-pqs8k" Jul 06 20:35:10.393114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2fb9f72c-98b1-4441-9c7d-57263e220a21-kubelet-config\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:10.393114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252243 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/366ffea7-6944-47a4-9d15-bcf3aca0756c-ovnkube-config\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.393114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252258 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-socket-dir\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.393114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-etc-selinux\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.393114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-sys-fs\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.393114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a989d7ea-35b8-47ea-8942-a5af5d9718ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.393114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a989d7ea-35b8-47ea-8942-a5af5d9718ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.393114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdskv\" (UniqueName: \"kubernetes.io/projected/a989d7ea-35b8-47ea-8942-a5af5d9718ab-kube-api-access-fdskv\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.393114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f0d0297-cff0-4b11-b42c-af7c14b392a4-host\") pod \"node-ca-whgpf\" (UID: \"6f0d0297-cff0-4b11-b42c-af7c14b392a4\") " pod="openshift-image-registry/node-ca-whgpf" Jul 06 20:35:10.393114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkr27\" (UniqueName: \"kubernetes.io/projected/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-kube-api-access-zkr27\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.393114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a989d7ea-35b8-47ea-8942-a5af5d9718ab-os-release\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.393114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252449 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-tuned\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.393114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-log-socket\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-run-openvswitch\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252557 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/366ffea7-6944-47a4-9d15-bcf3aca0756c-ovnkube-script-lib\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252576 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252579 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157e3ec8-eb18-430f-b4b5-397ab0281e2a-tmp-dir\") pod \"node-resolver-nsdj2\" (UID: \"157e3ec8-eb18-430f-b4b5-397ab0281e2a\") " pod="openshift-dns/node-resolver-nsdj2" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-sysconfig\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-var-lib-openvswitch\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.252672 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.254084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.254097 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.254869 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-splnw\"" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.254964 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.259268 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.259717 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gfwvs\"" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.275905 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.285407 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 20:30:09 +0000 UTC" deadline="2028-03-21 14:57:06.055904216 +0000 UTC" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:09.285447 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14970h21m56.770459208s" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.340597 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l8pcq" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.348975 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l8pcq" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.377117 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 20:35:10.393581 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.384658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.384678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a989d7ea-35b8-47ea-8942-a5af5d9718ab-system-cni-dir\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.384695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ba8e8ab-bd92-4f3e-9266-c41227aadedb-iptables-alerter-script\") pod \"iptables-alerter-6wgrb\" (UID: \"9ba8e8ab-bd92-4f3e-9266-c41227aadedb\") " pod="openshift-network-operator/iptables-alerter-6wgrb" Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.384716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-systemd-units\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.384733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-cni-bin\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.384803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-multus-socket-dir-parent\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:10.384809 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.384837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xwqp\" (UniqueName: \"kubernetes.io/projected/a01c55d5-4af0-4c27-8a7c-da00bae601c6-kube-api-access-9xwqp\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.384866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bcc5ed5-4a67-416b-bebf-93316f005012-agent-certs\") pod \"konnectivity-agent-pqs8k\" (UID: \"8bcc5ed5-4a67-416b-bebf-93316f005012\") " pod="kube-system/konnectivity-agent-pqs8k" Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:10.384881 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret podName:2fb9f72c-98b1-4441-9c7d-57263e220a21 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:10.884860986 +0000 UTC m=+2.015223578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret") pod "global-pull-secret-syncer-kfgw5" (UID: "2fb9f72c-98b1-4441-9c7d-57263e220a21") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.384912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2fb9f72c-98b1-4441-9c7d-57263e220a21-kubelet-config\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.384936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/366ffea7-6944-47a4-9d15-bcf3aca0756c-ovnkube-config\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.384940 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-cni-bin\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.384979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-run-k8s-cni-cncf-io\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.394154 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.384992 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-systemd-units\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-run-multus-certs\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385000 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2fb9f72c-98b1-4441-9c7d-57263e220a21-kubelet-config\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-socket-dir\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385051 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a989d7ea-35b8-47ea-8942-a5af5d9718ab-system-cni-dir\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-etc-selinux\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-sys-fs\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a989d7ea-35b8-47ea-8942-a5af5d9718ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a989d7ea-35b8-47ea-8942-a5af5d9718ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdskv\" (UniqueName: \"kubernetes.io/projected/a989d7ea-35b8-47ea-8942-a5af5d9718ab-kube-api-access-fdskv\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f0d0297-cff0-4b11-b42c-af7c14b392a4-host\") pod \"node-ca-whgpf\" (UID: \"6f0d0297-cff0-4b11-b42c-af7c14b392a4\") " pod="openshift-image-registry/node-ca-whgpf" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385175 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-socket-dir\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385177 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-sys-fs\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385203 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f0d0297-cff0-4b11-b42c-af7c14b392a4-host\") pod \"node-ca-whgpf\" (UID: \"6f0d0297-cff0-4b11-b42c-af7c14b392a4\") " pod="openshift-image-registry/node-ca-whgpf" Jul 06 20:35:10.394607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkr27\" (UniqueName: \"kubernetes.io/projected/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-kube-api-access-zkr27\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a989d7ea-35b8-47ea-8942-a5af5d9718ab-os-release\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-tuned\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385271 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-etc-selinux\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-log-socket\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385309 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a989d7ea-35b8-47ea-8942-a5af5d9718ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385326 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a989d7ea-35b8-47ea-8942-a5af5d9718ab-os-release\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385343 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-log-socket\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385349 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-system-cni-dir\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jm5\" (UniqueName: \"kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5\") pod \"network-check-target-m6hmf\" (UID: \"76348c50-5f18-4580-95e2-9a92aba127dc\") " pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-run-openvswitch\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/366ffea7-6944-47a4-9d15-bcf3aca0756c-ovnkube-script-lib\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.395022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-run-netns\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-var-lib-cni-bin\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-var-lib-cni-multus\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157e3ec8-eb18-430f-b4b5-397ab0281e2a-tmp-dir\") pod \"node-resolver-nsdj2\" (UID: \"157e3ec8-eb18-430f-b4b5-397ab0281e2a\") " pod="openshift-dns/node-resolver-nsdj2" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-sysconfig\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385792 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-var-lib-openvswitch\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-multus-cni-dir\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxghq\" (UniqueName: \"kubernetes.io/projected/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-kube-api-access-vxghq\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljv69\" (UniqueName: \"kubernetes.io/projected/157e3ec8-eb18-430f-b4b5-397ab0281e2a-kube-api-access-ljv69\") pod \"node-resolver-nsdj2\" (UID: \"157e3ec8-eb18-430f-b4b5-397ab0281e2a\") " pod="openshift-dns/node-resolver-nsdj2" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385873 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a989d7ea-35b8-47ea-8942-a5af5d9718ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385625 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385937 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-run-openvswitch\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.385994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a989d7ea-35b8-47ea-8942-a5af5d9718ab-cnibin\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-sys\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386043 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a01c55d5-4af0-4c27-8a7c-da00bae601c6-cni-binary-copy\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.395521 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386064 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-hostroot\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386077 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ba8e8ab-bd92-4f3e-9266-c41227aadedb-iptables-alerter-script\") pod \"iptables-alerter-6wgrb\" (UID: \"9ba8e8ab-bd92-4f3e-9266-c41227aadedb\") " pod="openshift-network-operator/iptables-alerter-6wgrb" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-cnibin\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386161 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-sysconfig\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386205 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a989d7ea-35b8-47ea-8942-a5af5d9718ab-cnibin\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386224 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-sys\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-etc-kubernetes\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/157e3ec8-eb18-430f-b4b5-397ab0281e2a-hosts-file\") pod \"node-resolver-nsdj2\" (UID: \"157e3ec8-eb18-430f-b4b5-397ab0281e2a\") " pod="openshift-dns/node-resolver-nsdj2" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386379 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/157e3ec8-eb18-430f-b4b5-397ab0281e2a-tmp-dir\") pod \"node-resolver-nsdj2\" (UID: \"157e3ec8-eb18-430f-b4b5-397ab0281e2a\") " pod="openshift-dns/node-resolver-nsdj2" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-registration-dir\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-registration-dir\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386424 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-slash\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386452 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-run-netns\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.395976 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386459 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/157e3ec8-eb18-430f-b4b5-397ab0281e2a-hosts-file\") pod \"node-resolver-nsdj2\" (UID: \"157e3ec8-eb18-430f-b4b5-397ab0281e2a\") " pod="openshift-dns/node-resolver-nsdj2" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a989d7ea-35b8-47ea-8942-a5af5d9718ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-kubernetes\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386508 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/366ffea7-6944-47a4-9d15-bcf3aca0756c-ovnkube-config\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386587 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-kubernetes\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386511 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-var-lib-openvswitch\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-sysctl-conf\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386641 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-sysctl-conf\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386658 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/366ffea7-6944-47a4-9d15-bcf3aca0756c-ovnkube-script-lib\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386691 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-slash\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-run\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386734 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-run-netns\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f0d0297-cff0-4b11-b42c-af7c14b392a4-serviceca\") pod \"node-ca-whgpf\" (UID: \"6f0d0297-cff0-4b11-b42c-af7c14b392a4\") " pod="openshift-image-registry/node-ca-whgpf" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-kubelet\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386894 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-run\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.396421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386939 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-kubelet\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.386990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-sysctl-d\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-systemd\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-lib-modules\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-os-release\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-run-ovn\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-device-dir\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-var-lib-kubelet\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-node-log\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ba8e8ab-bd92-4f3e-9266-c41227aadedb-host-slash\") pod \"iptables-alerter-6wgrb\" (UID: \"9ba8e8ab-bd92-4f3e-9266-c41227aadedb\") " pod="openshift-network-operator/iptables-alerter-6wgrb" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387213 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-device-dir\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-var-lib-kubelet\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387168 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a989d7ea-35b8-47ea-8942-a5af5d9718ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.396860 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387260 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-systemd\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387304 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-sysctl-d\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387315 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387337 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-node-log\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387320 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f0d0297-cff0-4b11-b42c-af7c14b392a4-serviceca\") pod \"node-ca-whgpf\" (UID: \"6f0d0297-cff0-4b11-b42c-af7c14b392a4\") " pod="openshift-image-registry/node-ca-whgpf" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387369 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-run-ovn\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387380 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ba8e8ab-bd92-4f3e-9266-c41227aadedb-host-slash\") pod \"iptables-alerter-6wgrb\" (UID: \"9ba8e8ab-bd92-4f3e-9266-c41227aadedb\") " pod="openshift-network-operator/iptables-alerter-6wgrb" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387405 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-lib-modules\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387417 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-var-lib-kubelet\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-host\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81dde24b-8630-4c06-8ce9-542aa3db5276-tmp\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-host\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf54w\" (UniqueName: \"kubernetes.io/projected/81dde24b-8630-4c06-8ce9-542aa3db5276-kube-api-access-qf54w\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-run-systemd\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a989d7ea-35b8-47ea-8942-a5af5d9718ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.397321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-modprobe-d\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387641 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-run-systemd\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-etc-openvswitch\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-cni-netd\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/366ffea7-6944-47a4-9d15-bcf3aca0756c-env-overrides\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387744 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-modprobe-d\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qrzz\" (UniqueName: \"kubernetes.io/projected/366ffea7-6944-47a4-9d15-bcf3aca0756c-kube-api-access-4qrzz\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-multus-conf-dir\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bcc5ed5-4a67-416b-bebf-93316f005012-konnectivity-ca\") pod \"konnectivity-agent-pqs8k\" (UID: \"8bcc5ed5-4a67-416b-bebf-93316f005012\") " pod="kube-system/konnectivity-agent-pqs8k" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77j94\" (UniqueName: \"kubernetes.io/projected/6f0d0297-cff0-4b11-b42c-af7c14b392a4-kube-api-access-77j94\") pod \"node-ca-whgpf\" (UID: \"6f0d0297-cff0-4b11-b42c-af7c14b392a4\") " pod="openshift-image-registry/node-ca-whgpf" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a01c55d5-4af0-4c27-8a7c-da00bae601c6-multus-daemon-config\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2fb9f72c-98b1-4441-9c7d-57263e220a21-dbus\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcwp9\" (UniqueName: \"kubernetes.io/projected/9ba8e8ab-bd92-4f3e-9266-c41227aadedb-kube-api-access-rcwp9\") pod \"iptables-alerter-6wgrb\" (UID: \"9ba8e8ab-bd92-4f3e-9266-c41227aadedb\") " pod="openshift-network-operator/iptables-alerter-6wgrb" Jul 06 20:35:10.397771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/366ffea7-6944-47a4-9d15-bcf3aca0756c-ovn-node-metrics-cert\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.398377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.387990 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-etc-openvswitch\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.398377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.388039 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-cni-netd\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.398377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.388069 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a989d7ea-35b8-47ea-8942-a5af5d9718ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.398377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.388072 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/366ffea7-6944-47a4-9d15-bcf3aca0756c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.398377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.388335 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2fb9f72c-98b1-4441-9c7d-57263e220a21-dbus\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:10.398377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.388797 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bcc5ed5-4a67-416b-bebf-93316f005012-konnectivity-ca\") pod \"konnectivity-agent-pqs8k\" (UID: \"8bcc5ed5-4a67-416b-bebf-93316f005012\") " pod="kube-system/konnectivity-agent-pqs8k" Jul 06 20:35:10.398377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.389020 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/366ffea7-6944-47a4-9d15-bcf3aca0756c-env-overrides\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.398377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.389366 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81dde24b-8630-4c06-8ce9-542aa3db5276-etc-tuned\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.398377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.389554 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bcc5ed5-4a67-416b-bebf-93316f005012-agent-certs\") pod \"konnectivity-agent-pqs8k\" (UID: \"8bcc5ed5-4a67-416b-bebf-93316f005012\") " pod="kube-system/konnectivity-agent-pqs8k" Jul 06 20:35:10.398377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.389570 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81dde24b-8630-4c06-8ce9-542aa3db5276-tmp\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.398377 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.390111 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/366ffea7-6944-47a4-9d15-bcf3aca0756c-ovn-node-metrics-cert\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.401661 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.400974 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf54w\" (UniqueName: \"kubernetes.io/projected/81dde24b-8630-4c06-8ce9-542aa3db5276-kube-api-access-qf54w\") pod \"tuned-zwvpx\" (UID: \"81dde24b-8630-4c06-8ce9-542aa3db5276\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.402465 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77j94\" (UniqueName: \"kubernetes.io/projected/6f0d0297-cff0-4b11-b42c-af7c14b392a4-kube-api-access-77j94\") pod \"node-ca-whgpf\" (UID: \"6f0d0297-cff0-4b11-b42c-af7c14b392a4\") " pod="openshift-image-registry/node-ca-whgpf" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.408421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-whgpf" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.408825 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkr27\" (UniqueName: \"kubernetes.io/projected/19bcd503-e2c9-4e99-a7b8-53a70ea3311f-kube-api-access-zkr27\") pod \"aws-ebs-csi-driver-node-t6bc5\" (UID: \"19bcd503-e2c9-4e99-a7b8-53a70ea3311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.410805 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdskv\" (UniqueName: \"kubernetes.io/projected/a989d7ea-35b8-47ea-8942-a5af5d9718ab-kube-api-access-fdskv\") pod \"multus-additional-cni-plugins-9frbs\" (UID: \"a989d7ea-35b8-47ea-8942-a5af5d9718ab\") " pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.410944 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcwp9\" (UniqueName: \"kubernetes.io/projected/9ba8e8ab-bd92-4f3e-9266-c41227aadedb-kube-api-access-rcwp9\") pod \"iptables-alerter-6wgrb\" (UID: \"9ba8e8ab-bd92-4f3e-9266-c41227aadedb\") " pod="openshift-network-operator/iptables-alerter-6wgrb" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.411608 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljv69\" (UniqueName: \"kubernetes.io/projected/157e3ec8-eb18-430f-b4b5-397ab0281e2a-kube-api-access-ljv69\") pod \"node-resolver-nsdj2\" (UID: \"157e3ec8-eb18-430f-b4b5-397ab0281e2a\") " pod="openshift-dns/node-resolver-nsdj2" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.412022 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qrzz\" (UniqueName: \"kubernetes.io/projected/366ffea7-6944-47a4-9d15-bcf3aca0756c-kube-api-access-4qrzz\") pod \"ovnkube-node-kqkjl\" (UID: \"366ffea7-6944-47a4-9d15-bcf3aca0756c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.434769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pqs8k" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.434818 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.488492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-multus-socket-dir-parent\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.488521 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xwqp\" (UniqueName: \"kubernetes.io/projected/a01c55d5-4af0-4c27-8a7c-da00bae601c6-kube-api-access-9xwqp\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.488588 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-run-k8s-cni-cncf-io\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.488606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-run-multus-certs\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.488635 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-system-cni-dir\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.488664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jm5\" (UniqueName: \"kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5\") pod \"network-check-target-m6hmf\" (UID: \"76348c50-5f18-4580-95e2-9a92aba127dc\") " pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:10.560783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.488695 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-run-multus-certs\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.488714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-run-netns\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.488731 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-system-cni-dir\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.488832 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-var-lib-cni-bin\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.488861 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-run-netns\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.488886 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-multus-socket-dir-parent\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.488941 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-run-k8s-cni-cncf-io\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-var-lib-cni-bin\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-var-lib-cni-multus\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-multus-cni-dir\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxghq\" (UniqueName: \"kubernetes.io/projected/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-kube-api-access-vxghq\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a01c55d5-4af0-4c27-8a7c-da00bae601c6-cni-binary-copy\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-hostroot\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-cnibin\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489280 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-multus-cni-dir\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.561763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489325 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-var-lib-cni-multus\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-etc-kubernetes\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489462 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-cnibin\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489530 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-etc-kubernetes\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-os-release\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489543 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-hostroot\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-var-lib-kubelet\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-multus-conf-dir\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489684 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-os-release\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489699 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-host-var-lib-kubelet\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.489751 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a01c55d5-4af0-4c27-8a7c-da00bae601c6-multus-conf-dir\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.490016 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a01c55d5-4af0-4c27-8a7c-da00bae601c6-cni-binary-copy\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:10.490117 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.490181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a01c55d5-4af0-4c27-8a7c-da00bae601c6-multus-daemon-config\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.564938 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:10.490237 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs podName:47c83a08-f54e-44ff-aaf6-cd03655c8b1e nodeName:}" failed. No retries permitted until 2026-07-06 20:35:10.990219047 +0000 UTC m=+2.120581619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs") pod "network-metrics-daemon-c5bs6" (UID: "47c83a08-f54e-44ff-aaf6-cd03655c8b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:10.565478 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.490730 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a01c55d5-4af0-4c27-8a7c-da00bae601c6-multus-daemon-config\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.565478 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:10.509406 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:35:10.565478 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:10.509432 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:35:10.565478 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:10.509444 2579 projected.go:196] Error preparing data for projected volume kube-api-access-k4jm5 for pod openshift-network-diagnostics/network-check-target-m6hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:10.565478 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:10.509491 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5 podName:76348c50-5f18-4580-95e2-9a92aba127dc nodeName:}" failed. No retries permitted until 2026-07-06 20:35:11.009478378 +0000 UTC m=+2.139840952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jm5" (UniqueName: "kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5") pod "network-check-target-m6hmf" (UID: "76348c50-5f18-4580-95e2-9a92aba127dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:10.565478 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.511605 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xwqp\" (UniqueName: \"kubernetes.io/projected/a01c55d5-4af0-4c27-8a7c-da00bae601c6-kube-api-access-9xwqp\") pod \"multus-g5469\" (UID: \"a01c55d5-4af0-4c27-8a7c-da00bae601c6\") " pod="openshift-multus/multus-g5469" Jul 06 20:35:10.565478 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.511643 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxghq\" (UniqueName: \"kubernetes.io/projected/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-kube-api-access-vxghq\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:10.695563 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.606579 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9frbs" Jul 06 20:35:10.695563 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.624155 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6wgrb" Jul 06 20:35:10.695563 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.640997 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" Jul 06 20:35:10.695563 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.644896 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" Jul 06 20:35:10.695563 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.666425 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nsdj2" Jul 06 20:35:10.842949 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:10.703498 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d37fda4a913c36c0e6c4c580018d46.slice/crio-b2486a89185f93668500041c3c9693e8148322f220d1c7bdef1569fb6e07eb20 WatchSource:0}: Error finding container b2486a89185f93668500041c3c9693e8148322f220d1c7bdef1569fb6e07eb20: Status 404 returned error can't find the container with id b2486a89185f93668500041c3c9693e8148322f220d1c7bdef1569fb6e07eb20 Jul 06 20:35:10.842949 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:10.708348 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9517028d6d269b9d2b51313aaffc1f4e.slice/crio-55a4e1a50f8c754d801ddf07f16f4b7e9eb30d4ff77454e35e44b93a29f5e08f WatchSource:0}: Error finding container 55a4e1a50f8c754d801ddf07f16f4b7e9eb30d4ff77454e35e44b93a29f5e08f: Status 404 returned error can't find the container with id 55a4e1a50f8c754d801ddf07f16f4b7e9eb30d4ff77454e35e44b93a29f5e08f Jul 06 20:35:10.842949 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.708381 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:35:10.842949 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.741452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g5469" Jul 06 20:35:10.842949 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:10.749474 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01c55d5_4af0_4c27_8a7c_da00bae601c6.slice/crio-86da2cece002b5add731fc3d8a7a5c68febae3dd4aad6c53bb9884910f75d9e2 WatchSource:0}: Error finding container 86da2cece002b5add731fc3d8a7a5c68febae3dd4aad6c53bb9884910f75d9e2: Status 404 returned error can't find the container with id 86da2cece002b5add731fc3d8a7a5c68febae3dd4aad6c53bb9884910f75d9e2 Jul 06 20:35:10.842949 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.799815 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:35:10.842949 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:10.815397 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcc5ed5_4a67_416b_bebf_93316f005012.slice/crio-8eca3dd9c51fdf5f3d2c10a7f63003bb9ff71b8587d8e1ac6b082f302d8be351 WatchSource:0}: Error finding container 8eca3dd9c51fdf5f3d2c10a7f63003bb9ff71b8587d8e1ac6b082f302d8be351: Status 404 returned error can't find the container with id 8eca3dd9c51fdf5f3d2c10a7f63003bb9ff71b8587d8e1ac6b082f302d8be351 Jul 06 20:35:10.989967 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.891801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:10.989967 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:10.891932 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:10.989967 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:10.891997 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret podName:2fb9f72c-98b1-4441-9c7d-57263e220a21 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:11.891983202 +0000 UTC m=+3.022345774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret") pod "global-pull-secret-syncer-kfgw5" (UID: "2fb9f72c-98b1-4441-9c7d-57263e220a21") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:10.989967 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:10.936122 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0d0297_cff0_4b11_b42c_af7c14b392a4.slice/crio-c45fe7afc0923d95971fb454912c2587246f89158171ed41b7e238eec67597a1 WatchSource:0}: Error finding container c45fe7afc0923d95971fb454912c2587246f89158171ed41b7e238eec67597a1: Status 404 returned error can't find the container with id c45fe7afc0923d95971fb454912c2587246f89158171ed41b7e238eec67597a1 Jul 06 20:35:10.992848 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:10.992763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:10.993001 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:10.992858 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:10.993001 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:10.992902 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs podName:47c83a08-f54e-44ff-aaf6-cd03655c8b1e nodeName:}" failed. No retries permitted until 2026-07-06 20:35:11.992890204 +0000 UTC m=+3.123252776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs") pod "network-metrics-daemon-c5bs6" (UID: "47c83a08-f54e-44ff-aaf6-cd03655c8b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:11.112632 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:11.044142 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda989d7ea_35b8_47ea_8942_a5af5d9718ab.slice/crio-790bd823acc235e3ced21b1eb12b0a1747fe36d815edf8fd5d26ccf8b2731375 WatchSource:0}: Error finding container 790bd823acc235e3ced21b1eb12b0a1747fe36d815edf8fd5d26ccf8b2731375: Status 404 returned error can't find the container with id 790bd823acc235e3ced21b1eb12b0a1747fe36d815edf8fd5d26ccf8b2731375 Jul 06 20:35:11.112632 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:11.046612 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366ffea7_6944_47a4_9d15_bcf3aca0756c.slice/crio-f2b6730bab569581c62d4a63e427c18104aba7731b784d12413441105a0a0811 WatchSource:0}: Error finding container f2b6730bab569581c62d4a63e427c18104aba7731b784d12413441105a0a0811: Status 404 returned error can't find the container with id f2b6730bab569581c62d4a63e427c18104aba7731b784d12413441105a0a0811 Jul 06 20:35:11.112632 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:11.047201 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba8e8ab_bd92_4f3e_9266_c41227aadedb.slice/crio-0a241c7e70cc0ecb8b5ec2abdb88a9d16ae1782c826b697cab34f4713e793c08 WatchSource:0}: Error finding container 0a241c7e70cc0ecb8b5ec2abdb88a9d16ae1782c826b697cab34f4713e793c08: Status 404 returned error can't find the container with id 0a241c7e70cc0ecb8b5ec2abdb88a9d16ae1782c826b697cab34f4713e793c08 Jul 06 20:35:11.112632 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:11.072459 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157e3ec8_eb18_430f_b4b5_397ab0281e2a.slice/crio-86a6fe178e52ed2b9c3c1987f991a4c4d92b0bd2409c672a97d7006999b317ba WatchSource:0}: Error finding container 86a6fe178e52ed2b9c3c1987f991a4c4d92b0bd2409c672a97d7006999b317ba: Status 404 returned error can't find the container with id 86a6fe178e52ed2b9c3c1987f991a4c4d92b0bd2409c672a97d7006999b317ba Jul 06 20:35:11.112632 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.093756 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jm5\" (UniqueName: \"kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5\") pod \"network-check-target-m6hmf\" (UID: \"76348c50-5f18-4580-95e2-9a92aba127dc\") " pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:11.112632 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:11.093928 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:35:11.112632 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:11.093948 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:35:11.112632 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:11.093975 2579 projected.go:196] Error preparing data for projected volume kube-api-access-k4jm5 for pod openshift-network-diagnostics/network-check-target-m6hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:11.112632 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:11.094032 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5 podName:76348c50-5f18-4580-95e2-9a92aba127dc nodeName:}" failed. No retries permitted until 2026-07-06 20:35:12.094013698 +0000 UTC m=+3.224376286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jm5" (UniqueName: "kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5") pod "network-check-target-m6hmf" (UID: "76348c50-5f18-4580-95e2-9a92aba127dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:11.112632 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:11.099692 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81dde24b_8630_4c06_8ce9_542aa3db5276.slice/crio-e31875975e38c02ef0fc6a49acb64eff044071da8310a8e1359ed224d2a6da0b WatchSource:0}: Error finding container e31875975e38c02ef0fc6a49acb64eff044071da8310a8e1359ed224d2a6da0b: Status 404 returned error can't find the container with id e31875975e38c02ef0fc6a49acb64eff044071da8310a8e1359ed224d2a6da0b Jul 06 20:35:11.272989 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.251684 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:35:11.289650 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:11.289049 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bcd503_e2c9_4e99_a7b8_53a70ea3311f.slice/crio-b81ecddbc00a8bc97ad9d5d2c040bc7f653967cff39eb93b7c05e51e3572cc94 WatchSource:0}: Error finding container b81ecddbc00a8bc97ad9d5d2c040bc7f653967cff39eb93b7c05e51e3572cc94: Status 404 returned error can't find the container with id b81ecddbc00a8bc97ad9d5d2c040bc7f653967cff39eb93b7c05e51e3572cc94 Jul 06 20:35:11.379396 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.358759 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:30:10 +0000 UTC" deadline="2028-01-18 23:37:08.290191646 +0000 UTC" Jul 06 20:35:11.379396 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.358792 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13467h1m56.931404252s" Jul 06 20:35:11.581922 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.581887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:11.582640 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:11.582039 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:11.582640 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.582437 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:11.582640 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:11.582531 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:11.590069 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.589390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" event={"ID":"9517028d6d269b9d2b51313aaffc1f4e","Type":"ContainerStarted","Data":"55a4e1a50f8c754d801ddf07f16f4b7e9eb30d4ff77454e35e44b93a29f5e08f"} Jul 06 20:35:11.593993 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.593907 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" event={"ID":"75d37fda4a913c36c0e6c4c580018d46","Type":"ContainerStarted","Data":"b2486a89185f93668500041c3c9693e8148322f220d1c7bdef1569fb6e07eb20"} Jul 06 20:35:11.601098 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.601025 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" event={"ID":"19bcd503-e2c9-4e99-a7b8-53a70ea3311f","Type":"ContainerStarted","Data":"b81ecddbc00a8bc97ad9d5d2c040bc7f653967cff39eb93b7c05e51e3572cc94"} Jul 06 20:35:11.607141 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.607073 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" event={"ID":"81dde24b-8630-4c06-8ce9-542aa3db5276","Type":"ContainerStarted","Data":"e31875975e38c02ef0fc6a49acb64eff044071da8310a8e1359ed224d2a6da0b"} Jul 06 20:35:11.609797 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.609770 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6wgrb" event={"ID":"9ba8e8ab-bd92-4f3e-9266-c41227aadedb","Type":"ContainerStarted","Data":"0a241c7e70cc0ecb8b5ec2abdb88a9d16ae1782c826b697cab34f4713e793c08"} Jul 06 20:35:11.613429 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.613402 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" event={"ID":"366ffea7-6944-47a4-9d15-bcf3aca0756c","Type":"ContainerStarted","Data":"f2b6730bab569581c62d4a63e427c18104aba7731b784d12413441105a0a0811"} Jul 06 20:35:11.622060 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.622033 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9frbs" event={"ID":"a989d7ea-35b8-47ea-8942-a5af5d9718ab","Type":"ContainerStarted","Data":"790bd823acc235e3ced21b1eb12b0a1747fe36d815edf8fd5d26ccf8b2731375"} Jul 06 20:35:11.630407 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.630346 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g5469" event={"ID":"a01c55d5-4af0-4c27-8a7c-da00bae601c6","Type":"ContainerStarted","Data":"86da2cece002b5add731fc3d8a7a5c68febae3dd4aad6c53bb9884910f75d9e2"} Jul 06 20:35:11.640843 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.640815 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nsdj2" event={"ID":"157e3ec8-eb18-430f-b4b5-397ab0281e2a","Type":"ContainerStarted","Data":"86a6fe178e52ed2b9c3c1987f991a4c4d92b0bd2409c672a97d7006999b317ba"} Jul 06 20:35:11.651899 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.651871 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-whgpf" event={"ID":"6f0d0297-cff0-4b11-b42c-af7c14b392a4","Type":"ContainerStarted","Data":"c45fe7afc0923d95971fb454912c2587246f89158171ed41b7e238eec67597a1"} Jul 06 20:35:11.660073 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.659999 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pqs8k" event={"ID":"8bcc5ed5-4a67-416b-bebf-93316f005012","Type":"ContainerStarted","Data":"8eca3dd9c51fdf5f3d2c10a7f63003bb9ff71b8587d8e1ac6b082f302d8be351"} Jul 06 20:35:11.904374 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:11.903657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:11.904374 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:11.903845 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:11.904374 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:11.903912 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret podName:2fb9f72c-98b1-4441-9c7d-57263e220a21 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:13.903892058 +0000 UTC m=+5.034254650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret") pod "global-pull-secret-syncer-kfgw5" (UID: "2fb9f72c-98b1-4441-9c7d-57263e220a21") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:12.004926 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:12.004893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:12.005160 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:12.005056 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:12.005160 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:12.005115 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs podName:47c83a08-f54e-44ff-aaf6-cd03655c8b1e nodeName:}" failed. No retries permitted until 2026-07-06 20:35:14.005096283 +0000 UTC m=+5.135458860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs") pod "network-metrics-daemon-c5bs6" (UID: "47c83a08-f54e-44ff-aaf6-cd03655c8b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:12.105917 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:12.105317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jm5\" (UniqueName: \"kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5\") pod \"network-check-target-m6hmf\" (UID: \"76348c50-5f18-4580-95e2-9a92aba127dc\") " pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:12.105917 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:12.105468 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:35:12.105917 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:12.105489 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:35:12.105917 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:12.105502 2579 projected.go:196] Error preparing data for projected volume kube-api-access-k4jm5 for pod openshift-network-diagnostics/network-check-target-m6hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:12.105917 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:12.105558 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5 podName:76348c50-5f18-4580-95e2-9a92aba127dc nodeName:}" failed. No retries permitted until 2026-07-06 20:35:14.105539937 +0000 UTC m=+5.235902514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jm5" (UniqueName: "kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5") pod "network-check-target-m6hmf" (UID: "76348c50-5f18-4580-95e2-9a92aba127dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:12.359223 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:12.359130 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:30:10 +0000 UTC" deadline="2028-04-01 22:50:11.524356352 +0000 UTC" Jul 06 20:35:12.359223 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:12.359189 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15242h14m59.165171037s" Jul 06 20:35:12.364031 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:12.363811 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:35:12.580026 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:12.579526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:12.580026 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:12.579666 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:13.580146 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:13.580114 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:13.580714 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:13.580266 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:13.580714 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:13.580669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:13.580829 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:13.580769 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:13.919972 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:13.919916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:13.920187 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:13.920136 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:13.920248 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:13.920212 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret podName:2fb9f72c-98b1-4441-9c7d-57263e220a21 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:17.920191329 +0000 UTC m=+9.050553923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret") pod "global-pull-secret-syncer-kfgw5" (UID: "2fb9f72c-98b1-4441-9c7d-57263e220a21") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:14.020630 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:14.020587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:14.020793 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:14.020757 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:14.020861 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:14.020827 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs podName:47c83a08-f54e-44ff-aaf6-cd03655c8b1e nodeName:}" failed. No retries permitted until 2026-07-06 20:35:18.020806463 +0000 UTC m=+9.151169034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs") pod "network-metrics-daemon-c5bs6" (UID: "47c83a08-f54e-44ff-aaf6-cd03655c8b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:14.121893 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:14.121857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jm5\" (UniqueName: \"kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5\") pod \"network-check-target-m6hmf\" (UID: \"76348c50-5f18-4580-95e2-9a92aba127dc\") " pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:14.122209 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:14.122166 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:35:14.122290 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:14.122233 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:35:14.122290 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:14.122273 2579 projected.go:196] Error preparing data for projected volume kube-api-access-k4jm5 for pod openshift-network-diagnostics/network-check-target-m6hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:14.122387 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:14.122373 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5 podName:76348c50-5f18-4580-95e2-9a92aba127dc nodeName:}" failed. No retries permitted until 2026-07-06 20:35:18.122353959 +0000 UTC m=+9.252716556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jm5" (UniqueName: "kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5") pod "network-check-target-m6hmf" (UID: "76348c50-5f18-4580-95e2-9a92aba127dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:14.578894 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:14.578860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:14.579056 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:14.579032 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:15.578875 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:15.578844 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:15.579366 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:15.578991 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:15.579366 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:15.579321 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:15.579478 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:15.579409 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:16.579132 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:16.579095 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:16.579584 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:16.579246 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:17.580293 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:17.580232 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:17.580722 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:17.580373 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:17.580722 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:17.580427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:17.580722 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:17.580539 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:17.960358 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:17.960280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:17.960499 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:17.960457 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:17.960553 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:17.960531 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret podName:2fb9f72c-98b1-4441-9c7d-57263e220a21 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:25.960510549 +0000 UTC m=+17.090873123 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret") pod "global-pull-secret-syncer-kfgw5" (UID: "2fb9f72c-98b1-4441-9c7d-57263e220a21") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:18.061058 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:18.060942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:18.061236 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:18.061114 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:18.061236 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:18.061189 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs podName:47c83a08-f54e-44ff-aaf6-cd03655c8b1e nodeName:}" failed. No retries permitted until 2026-07-06 20:35:26.061168085 +0000 UTC m=+17.191530669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs") pod "network-metrics-daemon-c5bs6" (UID: "47c83a08-f54e-44ff-aaf6-cd03655c8b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:18.162264 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:18.162109 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jm5\" (UniqueName: \"kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5\") pod \"network-check-target-m6hmf\" (UID: \"76348c50-5f18-4580-95e2-9a92aba127dc\") " pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:18.162264 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:18.162264 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:35:18.162493 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:18.162284 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:35:18.162493 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:18.162296 2579 projected.go:196] Error preparing data for projected volume kube-api-access-k4jm5 for pod openshift-network-diagnostics/network-check-target-m6hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:18.162493 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:18.162353 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5 podName:76348c50-5f18-4580-95e2-9a92aba127dc nodeName:}" failed. No retries permitted until 2026-07-06 20:35:26.162333911 +0000 UTC m=+17.292696504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jm5" (UniqueName: "kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5") pod "network-check-target-m6hmf" (UID: "76348c50-5f18-4580-95e2-9a92aba127dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:18.579291 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:18.579254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:18.579448 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:18.579405 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:19.580018 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:19.579984 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:19.580484 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:19.580104 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:19.580484 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:19.579996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:19.580484 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:19.580191 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:20.579066 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:20.579032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:20.579244 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:20.579177 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:21.579371 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:21.579325 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:21.579371 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:21.579339 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:21.579881 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:21.579480 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:21.579881 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:21.579600 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:22.579125 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:22.579089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:22.579294 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:22.579208 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:23.578677 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:23.578641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:23.579147 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:23.578641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:23.579147 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:23.578767 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:23.579147 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:23.578870 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:24.579299 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:24.579269 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:24.579705 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:24.579384 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:25.578652 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:25.578570 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:25.578652 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:25.578609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:25.578858 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:25.578706 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:25.579073 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:25.579039 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:26.021986 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:26.021931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:26.022438 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:26.022107 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:26.022438 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:26.022188 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret podName:2fb9f72c-98b1-4441-9c7d-57263e220a21 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:42.022167129 +0000 UTC m=+33.152529702 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret") pod "global-pull-secret-syncer-kfgw5" (UID: "2fb9f72c-98b1-4441-9c7d-57263e220a21") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:26.123175 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:26.123130 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:26.123354 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:26.123301 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:26.123414 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:26.123385 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs podName:47c83a08-f54e-44ff-aaf6-cd03655c8b1e nodeName:}" failed. No retries permitted until 2026-07-06 20:35:42.1233634 +0000 UTC m=+33.253726038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs") pod "network-metrics-daemon-c5bs6" (UID: "47c83a08-f54e-44ff-aaf6-cd03655c8b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:26.224429 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:26.224391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jm5\" (UniqueName: \"kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5\") pod \"network-check-target-m6hmf\" (UID: \"76348c50-5f18-4580-95e2-9a92aba127dc\") " pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:26.224604 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:26.224572 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:35:26.224604 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:26.224597 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:35:26.224707 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:26.224612 2579 projected.go:196] Error preparing data for projected volume kube-api-access-k4jm5 for pod openshift-network-diagnostics/network-check-target-m6hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:26.224759 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:26.224672 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5 podName:76348c50-5f18-4580-95e2-9a92aba127dc nodeName:}" failed. No retries permitted until 2026-07-06 20:35:42.224653308 +0000 UTC m=+33.355015898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jm5" (UniqueName: "kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5") pod "network-check-target-m6hmf" (UID: "76348c50-5f18-4580-95e2-9a92aba127dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:26.579543 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:26.579513 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:26.579716 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:26.579636 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:27.578898 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:27.578862 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:27.578898 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:27.578887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:27.579416 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:27.579000 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:27.579416 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:27.579147 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:28.579033 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:28.578997 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:28.579381 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:28.579148 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:29.579923 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:29.579722 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:29.580275 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:29.579787 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:29.580275 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:29.580019 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:29.580275 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:29.580059 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:30.578874 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.578711 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:30.578988 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:30.578969 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:30.708509 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.708431 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" event={"ID":"81dde24b-8630-4c06-8ce9-542aa3db5276","Type":"ContainerStarted","Data":"da210a5fb9326d33640f5ee17e432f19eca4022eba69eaa4bf5ecb80253e31a4"} Jul 06 20:35:30.711218 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.711197 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 20:35:30.711576 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.711557 2579 generic.go:356] "Generic (PLEG): container finished" podID="366ffea7-6944-47a4-9d15-bcf3aca0756c" containerID="218d53b20949b66d215540b33b4eede9c0d6ca5f4dcb46b03f81721ac5a5d433" exitCode=1 Jul 06 20:35:30.711676 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.711627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" event={"ID":"366ffea7-6944-47a4-9d15-bcf3aca0756c","Type":"ContainerStarted","Data":"39d4d93cb476d0bb643915b8d149d2919d1ee3feb2ae02436ec6a45eaa02d6de"} Jul 06 20:35:30.711746 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.711690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" event={"ID":"366ffea7-6944-47a4-9d15-bcf3aca0756c","Type":"ContainerStarted","Data":"83b664301d8fafc94c8261aafcc33fa8a10bf2b055da285b259579208c5ae036"} Jul 06 20:35:30.711746 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.711728 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" event={"ID":"366ffea7-6944-47a4-9d15-bcf3aca0756c","Type":"ContainerStarted","Data":"438a1ed4ba92f837a631d38042cd028aab9c245c4b7ba40aa844b6b6d21afee7"} Jul 06 20:35:30.711837 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.711768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" event={"ID":"366ffea7-6944-47a4-9d15-bcf3aca0756c","Type":"ContainerStarted","Data":"b1462dca7ca7e1d3378e0be956ef7c12ec6476ff7f414fdb2622cac3c9079bb3"} Jul 06 20:35:30.711837 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.711806 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" event={"ID":"366ffea7-6944-47a4-9d15-bcf3aca0756c","Type":"ContainerDied","Data":"218d53b20949b66d215540b33b4eede9c0d6ca5f4dcb46b03f81721ac5a5d433"} Jul 06 20:35:30.711926 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.711845 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" event={"ID":"366ffea7-6944-47a4-9d15-bcf3aca0756c","Type":"ContainerStarted","Data":"2a431479297cd61c6124e47abb64d3ac9098b71c30aedf2b80f862c3220e7b1d"} Jul 06 20:35:30.713112 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.713092 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g5469" event={"ID":"a01c55d5-4af0-4c27-8a7c-da00bae601c6","Type":"ContainerStarted","Data":"1417fc0b0707f7a3acda9e6d66e113066c77d7acc9a83e2658d836ee86b4dd2c"} Jul 06 20:35:30.714403 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.714384 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" event={"ID":"9517028d6d269b9d2b51313aaffc1f4e","Type":"ContainerStarted","Data":"bf6d722063a8fcd00e8d8fd7e85471397e90aeef5b71c63d067a8374d2a638fb"} Jul 06 20:35:30.736940 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.736886 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zwvpx" podStartSLOduration=2.847310143 podStartE2EDuration="21.736875854s" totalImagesPullingTime="18.889565711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:30.736701233 +0000 UTC m=+21.867063826" watchObservedRunningTime="2026-07-06 20:35:30.736875854 +0000 UTC m=+21.867238447" Jul 06 20:35:30.757611 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.755996 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" podStartSLOduration=20.755980244 podStartE2EDuration="20.755980244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:30.755417217 +0000 UTC m=+21.885779810" watchObservedRunningTime="2026-07-06 20:35:30.755980244 +0000 UTC m=+21.886342839" Jul 06 20:35:30.787184 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:30.787151 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g5469" podStartSLOduration=2.43017357 podStartE2EDuration="21.787139748s" totalImagesPullingTime="19.356966178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:30.786809952 +0000 UTC m=+21.917172546" watchObservedRunningTime="2026-07-06 20:35:30.787139748 +0000 UTC m=+21.917502340" Jul 06 20:35:31.578974 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.578781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:31.579125 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.578844 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:31.579125 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:31.579056 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:31.579208 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:31.579125 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:31.718160 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.718096 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" event={"ID":"19bcd503-e2c9-4e99-a7b8-53a70ea3311f","Type":"ContainerStarted","Data":"543357c396a8c383ae7d2d97523d234849fa7e017b6b32ab99b6a390dc08933f"} Jul 06 20:35:31.719782 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.719750 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6wgrb" event={"ID":"9ba8e8ab-bd92-4f3e-9266-c41227aadedb","Type":"ContainerStarted","Data":"4c7203ffa4fbf771f9de3faa26d327e940c8bf7337110e33fe005d00e58931b6"} Jul 06 20:35:31.721733 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.721694 2579 generic.go:356] "Generic (PLEG): container finished" podID="a989d7ea-35b8-47ea-8942-a5af5d9718ab" containerID="d9a96660bab670d8f2f8e842df9156c0092a7c81cbab77875abdfafffc2c821f" exitCode=0 Jul 06 20:35:31.721868 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.721771 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9frbs" event={"ID":"a989d7ea-35b8-47ea-8942-a5af5d9718ab","Type":"ContainerDied","Data":"d9a96660bab670d8f2f8e842df9156c0092a7c81cbab77875abdfafffc2c821f"} Jul 06 20:35:31.723630 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.723582 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nsdj2" event={"ID":"157e3ec8-eb18-430f-b4b5-397ab0281e2a","Type":"ContainerStarted","Data":"fa0acf8b78bae1a3c25886dfb25958f7fbb4fb55facdda57f8213ba66ce59c21"} Jul 06 20:35:31.725340 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.725309 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-whgpf" event={"ID":"6f0d0297-cff0-4b11-b42c-af7c14b392a4","Type":"ContainerStarted","Data":"9ac0ff3425a9ea8d1baea3d2e7633f4074f329b1c49ae8d86400846b4ee60689"} Jul 06 20:35:31.726883 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.726855 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pqs8k" event={"ID":"8bcc5ed5-4a67-416b-bebf-93316f005012","Type":"ContainerStarted","Data":"02f7baea5d0ceaf3e2246ba8bbe5cef98bbbe8cd69bece4b9238b57ebb1b31bd"} Jul 06 20:35:31.728474 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.728453 2579 generic.go:356] "Generic (PLEG): container finished" podID="75d37fda4a913c36c0e6c4c580018d46" containerID="66730416ebfbc02cf6a083236ff52d9a7c390524408cc9c8c1e633dae1400c9b" exitCode=0 Jul 06 20:35:31.728573 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.728511 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" event={"ID":"75d37fda4a913c36c0e6c4c580018d46","Type":"ContainerDied","Data":"66730416ebfbc02cf6a083236ff52d9a7c390524408cc9c8c1e633dae1400c9b"} Jul 06 20:35:31.736819 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.736782 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6wgrb" podStartSLOduration=3.834169179 podStartE2EDuration="22.736769748s" totalImagesPullingTime="18.902600569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:31.736436939 +0000 UTC m=+22.866799531" watchObservedRunningTime="2026-07-06 20:35:31.736769748 +0000 UTC m=+22.867132340" Jul 06 20:35:31.794065 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.794011 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nsdj2" podStartSLOduration=3.898840154 podStartE2EDuration="22.793991627s" totalImagesPullingTime="18.895151473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:31.793787558 +0000 UTC m=+22.924150151" watchObservedRunningTime="2026-07-06 20:35:31.793991627 +0000 UTC m=+22.924354221" Jul 06 20:35:31.810720 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.810679 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-whgpf" podStartSLOduration=8.213996134 podStartE2EDuration="22.810666961s" totalImagesPullingTime="14.596670827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:31.810363178 +0000 UTC m=+22.940725773" watchObservedRunningTime="2026-07-06 20:35:31.810666961 +0000 UTC m=+22.941029553" Jul 06 20:35:31.825851 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:31.825810 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pqs8k" podStartSLOduration=3.691399353 podStartE2EDuration="22.82579273s" totalImagesPullingTime="19.134393377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:31.82562866 +0000 UTC m=+22.955991254" watchObservedRunningTime="2026-07-06 20:35:31.82579273 +0000 UTC m=+22.956155325" Jul 06 20:35:32.492845 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:32.492815 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 20:35:32.579235 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:32.579207 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:32.579396 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:32.579333 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:32.732804 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:32.732772 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" event={"ID":"19bcd503-e2c9-4e99-a7b8-53a70ea3311f","Type":"ContainerStarted","Data":"5b9a264af4df3e38a45dc71c8414321fb9d4ee70e820367c24d40207aeb948fb"} Jul 06 20:35:32.734684 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:32.734657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" event={"ID":"75d37fda4a913c36c0e6c4c580018d46","Type":"ContainerStarted","Data":"82e9f42e483b7e829ffe95c22f7ef6f317e2551badee0de78254992daa049f3b"} Jul 06 20:35:33.386396 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:33.386305 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T20:35:32.492838044Z","UUID":"3bd2d834-891e-41bc-8aba-068df33a2513","Handler":null,"Name":"","Endpoint":""} Jul 06 20:35:33.387891 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:33.387865 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 20:35:33.388032 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:33.387903 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 20:35:33.578806 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:33.578770 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:33.578928 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:33.578809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:33.578928 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:33.578897 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:33.579067 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:33.579030 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:33.739930 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:33.739841 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" event={"ID":"19bcd503-e2c9-4e99-a7b8-53a70ea3311f","Type":"ContainerStarted","Data":"b42fece6136d77c62a3cf97cda478e2ec7964736dac1a88ee2b86c3035b634b0"} Jul 06 20:35:33.743417 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:33.743390 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 20:35:33.743877 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:33.743850 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" event={"ID":"366ffea7-6944-47a4-9d15-bcf3aca0756c","Type":"ContainerStarted","Data":"241cc47568dd71381fd139da153b4bd130e94b219903fe928cba3390c3264713"} Jul 06 20:35:33.760524 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:33.760483 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t6bc5" podStartSLOduration=3.875141492 podStartE2EDuration="24.760469579s" totalImagesPullingTime="20.885328087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:33.760228115 +0000 UTC m=+24.890590707" watchObservedRunningTime="2026-07-06 20:35:33.760469579 +0000 UTC m=+24.890832172" Jul 06 20:35:33.760919 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:33.760887 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" podStartSLOduration=23.760876966 podStartE2EDuration="23.760876966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:32.753240113 +0000 UTC m=+23.883602713" watchObservedRunningTime="2026-07-06 20:35:33.760876966 +0000 UTC m=+24.891239559" Jul 06 20:35:34.579266 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:34.579228 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:34.579456 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:34.579374 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:35.579362 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:35.579322 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:35.579362 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:35.579348 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:35.579799 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:35.579447 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:35.579799 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:35.579559 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:35.977772 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:35.977554 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pqs8k" Jul 06 20:35:35.978150 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:35.978130 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pqs8k" Jul 06 20:35:36.579018 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:36.578994 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:36.579174 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:36.579104 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:36.752401 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:36.752375 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 20:35:36.753100 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:36.753079 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" event={"ID":"366ffea7-6944-47a4-9d15-bcf3aca0756c","Type":"ContainerStarted","Data":"fdd5c3d1b8bf963a5d7913e8106e9a52c16aaacc03a14928ef0ce1d71c61ddc3"} Jul 06 20:35:36.753293 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:36.753265 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:36.753513 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:36.753498 2579 scope.go:117] "RemoveContainer" containerID="218d53b20949b66d215540b33b4eede9c0d6ca5f4dcb46b03f81721ac5a5d433" Jul 06 20:35:36.755310 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:36.755289 2579 generic.go:356] "Generic (PLEG): container finished" podID="a989d7ea-35b8-47ea-8942-a5af5d9718ab" containerID="38a6206f513c6e227d1044f55777847ffad6e199585d4ed295026b1c94e8cf30" exitCode=0 Jul 06 20:35:36.755411 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:36.755355 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9frbs" event={"ID":"a989d7ea-35b8-47ea-8942-a5af5d9718ab","Type":"ContainerDied","Data":"38a6206f513c6e227d1044f55777847ffad6e199585d4ed295026b1c94e8cf30"} Jul 06 20:35:36.755659 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:36.755628 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pqs8k" Jul 06 20:35:36.756214 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:36.756122 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pqs8k" Jul 06 20:35:36.769792 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:36.769774 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:37.578910 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:37.578873 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:37.579150 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:37.579001 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:37.579150 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:37.579083 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:37.579270 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:37.579197 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:37.760505 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:37.760484 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 20:35:37.760874 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:37.760848 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" event={"ID":"366ffea7-6944-47a4-9d15-bcf3aca0756c","Type":"ContainerStarted","Data":"f7cfd1e8c883e4040af0362b9cd66cdf9d7d9131e7f732d17ca30d4d3223dfd0"} Jul 06 20:35:37.776412 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:37.776356 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:37.776526 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:37.776504 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:37.793470 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:37.793446 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:35:37.855632 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:37.855449 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" podStartSLOduration=9.867558788 podStartE2EDuration="28.855433274s" totalImagesPullingTime="18.987874486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:37.815766846 +0000 UTC m=+28.946129440" watchObservedRunningTime="2026-07-06 20:35:37.855433274 +0000 UTC m=+28.985795867" Jul 06 20:35:38.104497 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:38.103068 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m6hmf"] Jul 06 20:35:38.104497 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:38.103227 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:38.104497 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:38.103354 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:38.106713 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:38.106688 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kfgw5"] Jul 06 20:35:38.106831 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:38.106784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:38.106886 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:38.106869 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:38.116670 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:38.116641 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c5bs6"] Jul 06 20:35:38.116771 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:38.116758 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:38.116863 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:38.116846 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:38.764653 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:38.764620 2579 generic.go:356] "Generic (PLEG): container finished" podID="a989d7ea-35b8-47ea-8942-a5af5d9718ab" containerID="2e5ac6c4ba7649544967571d6a299e94fc03ec93c4a2abf7c291fea9619857e7" exitCode=0 Jul 06 20:35:38.765048 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:38.764709 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9frbs" event={"ID":"a989d7ea-35b8-47ea-8942-a5af5d9718ab","Type":"ContainerDied","Data":"2e5ac6c4ba7649544967571d6a299e94fc03ec93c4a2abf7c291fea9619857e7"} Jul 06 20:35:39.579915 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:39.579888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:39.580062 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:39.579996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:39.580062 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:39.580007 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:39.580157 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:39.580103 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:39.580157 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:39.580150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:39.580257 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:39.580238 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:39.769015 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:39.768982 2579 generic.go:356] "Generic (PLEG): container finished" podID="a989d7ea-35b8-47ea-8942-a5af5d9718ab" containerID="facb9c5dc5471de7ca7f0d9bbfb43c75723153854fbf76837d9baebbbbc59258" exitCode=0 Jul 06 20:35:39.769338 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:39.769046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9frbs" event={"ID":"a989d7ea-35b8-47ea-8942-a5af5d9718ab","Type":"ContainerDied","Data":"facb9c5dc5471de7ca7f0d9bbfb43c75723153854fbf76837d9baebbbbc59258"} Jul 06 20:35:41.578938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:41.578907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:41.579376 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:41.579032 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6hmf" podUID="76348c50-5f18-4580-95e2-9a92aba127dc" Jul 06 20:35:41.579376 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:41.578907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:41.579376 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:41.579062 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:41.579376 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:41.579126 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kfgw5" podUID="2fb9f72c-98b1-4441-9c7d-57263e220a21" Jul 06 20:35:41.579376 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:41.579230 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5bs6" podUID="47c83a08-f54e-44ff-aaf6-cd03655c8b1e" Jul 06 20:35:42.042076 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.042036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:42.042250 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:42.042192 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:42.042308 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:42.042269 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret podName:2fb9f72c-98b1-4441-9c7d-57263e220a21 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:14.042249632 +0000 UTC m=+65.172612203 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret") pod "global-pull-secret-syncer-kfgw5" (UID: "2fb9f72c-98b1-4441-9c7d-57263e220a21") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:42.143051 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.143025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:42.143223 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:42.143183 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:42.143291 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:42.143252 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs podName:47c83a08-f54e-44ff-aaf6-cd03655c8b1e nodeName:}" failed. No retries permitted until 2026-07-06 20:36:14.143233908 +0000 UTC m=+65.273596497 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs") pod "network-metrics-daemon-c5bs6" (UID: "47c83a08-f54e-44ff-aaf6-cd03655c8b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:42.244158 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.244125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jm5\" (UniqueName: \"kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5\") pod \"network-check-target-m6hmf\" (UID: \"76348c50-5f18-4580-95e2-9a92aba127dc\") " pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:42.244333 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:42.244258 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:35:42.244333 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:42.244281 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:35:42.244333 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:42.244293 2579 projected.go:196] Error preparing data for projected volume kube-api-access-k4jm5 for pod openshift-network-diagnostics/network-check-target-m6hmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:42.244453 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:42.244349 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5 podName:76348c50-5f18-4580-95e2-9a92aba127dc nodeName:}" failed. No retries permitted until 2026-07-06 20:36:14.244332992 +0000 UTC m=+65.374695564 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jm5" (UniqueName: "kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5") pod "network-check-target-m6hmf" (UID: "76348c50-5f18-4580-95e2-9a92aba127dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:42.702673 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.702639 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeReady" Jul 06 20:35:42.703104 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.702795 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 20:35:42.741333 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.741297 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5867df54f9-rhdnn"] Jul 06 20:35:42.743931 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.743900 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.745942 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.745922 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 20:35:42.746059 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.746036 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lqq25\"" Jul 06 20:35:42.746223 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.746174 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 20:35:42.746306 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.746258 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 20:35:42.754294 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.753884 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 20:35:42.754804 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.754778 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7swx2"] Jul 06 20:35:42.756667 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.756650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7swx2" Jul 06 20:35:42.758644 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.758624 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 20:35:42.758746 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.758665 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mszj5\"" Jul 06 20:35:42.758746 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.758739 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 20:35:42.759608 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.759592 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 20:35:42.761341 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.761323 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h7rh2"] Jul 06 20:35:42.764502 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.764485 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5867df54f9-rhdnn"] Jul 06 20:35:42.764589 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.764565 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:42.766606 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.766580 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 20:35:42.766699 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.766584 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 20:35:42.766699 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.766666 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67jtg\"" Jul 06 20:35:42.769769 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.769749 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7swx2"] Jul 06 20:35:42.775486 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.775446 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h7rh2"] Jul 06 20:35:42.848360 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-ca-trust-extracted\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.848533 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-installation-pull-secrets\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.848533 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7c05db2-dcf9-406f-a781-4ad013ae0dfc-cert\") pod \"ingress-canary-7swx2\" (UID: \"f7c05db2-dcf9-406f-a781-4ad013ae0dfc\") " pod="openshift-ingress-canary/ingress-canary-7swx2" Jul 06 20:35:42.848533 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848436 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.848533 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848472 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18749804-aff5-4f10-a336-4562c73aff48-config-volume\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:42.848533 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848505 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-image-registry-private-configuration\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.848774 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18749804-aff5-4f10-a336-4562c73aff48-tmp-dir\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:42.848774 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-certificates\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.848774 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdzbh\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-kube-api-access-xdzbh\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.848774 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:42.848774 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8l2w\" (UniqueName: \"kubernetes.io/projected/f7c05db2-dcf9-406f-a781-4ad013ae0dfc-kube-api-access-t8l2w\") pod \"ingress-canary-7swx2\" (UID: \"f7c05db2-dcf9-406f-a781-4ad013ae0dfc\") " pod="openshift-ingress-canary/ingress-canary-7swx2" Jul 06 20:35:42.848999 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-trusted-ca\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.848999 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-bound-sa-token\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.848999 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.848834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-776px\" (UniqueName: \"kubernetes.io/projected/18749804-aff5-4f10-a336-4562c73aff48-kube-api-access-776px\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:42.926193 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.926148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqs8k_8bcc5ed5-4a67-416b-bebf-93316f005012/konnectivity-agent/0.log" limit=1024 Jul 06 20:35:42.950136 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.950296 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18749804-aff5-4f10-a336-4562c73aff48-config-volume\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:42.950296 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-image-registry-private-configuration\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.950296 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18749804-aff5-4f10-a336-4562c73aff48-tmp-dir\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:42.950296 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:42.950232 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:35:42.950296 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:42.950253 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5867df54f9-rhdnn: secret "image-registry-tls" not found Jul 06 20:35:42.950528 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:42.950312 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls podName:3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:43.450290023 +0000 UTC m=+34.580652595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls") pod "image-registry-5867df54f9-rhdnn" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3") : secret "image-registry-tls" not found Jul 06 20:35:42.950528 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-certificates\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.950528 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzbh\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-kube-api-access-xdzbh\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.950528 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:42.950528 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8l2w\" (UniqueName: \"kubernetes.io/projected/f7c05db2-dcf9-406f-a781-4ad013ae0dfc-kube-api-access-t8l2w\") pod \"ingress-canary-7swx2\" (UID: \"f7c05db2-dcf9-406f-a781-4ad013ae0dfc\") " pod="openshift-ingress-canary/ingress-canary-7swx2" Jul 06 20:35:42.950528 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-trusted-ca\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.950761 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950542 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-bound-sa-token\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.950761 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-776px\" (UniqueName: \"kubernetes.io/projected/18749804-aff5-4f10-a336-4562c73aff48-kube-api-access-776px\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:42.950761 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-ca-trust-extracted\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.950761 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950608 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-installation-pull-secrets\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.950761 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.950639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7c05db2-dcf9-406f-a781-4ad013ae0dfc-cert\") pod \"ingress-canary-7swx2\" (UID: \"f7c05db2-dcf9-406f-a781-4ad013ae0dfc\") " pod="openshift-ingress-canary/ingress-canary-7swx2" Jul 06 20:35:42.951021 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:42.950896 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:35:42.951021 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:42.950935 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls podName:18749804-aff5-4f10-a336-4562c73aff48 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:43.450921711 +0000 UTC m=+34.581284282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls") pod "dns-default-h7rh2" (UID: "18749804-aff5-4f10-a336-4562c73aff48") : secret "dns-default-metrics-tls" not found Jul 06 20:35:42.951169 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.951126 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-certificates\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.951467 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.951441 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18749804-aff5-4f10-a336-4562c73aff48-tmp-dir\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:42.951983 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.951944 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18749804-aff5-4f10-a336-4562c73aff48-config-volume\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:42.952727 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.952397 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-trusted-ca\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.952727 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.952556 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-ca-trust-extracted\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.957080 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.957060 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7c05db2-dcf9-406f-a781-4ad013ae0dfc-cert\") pod \"ingress-canary-7swx2\" (UID: \"f7c05db2-dcf9-406f-a781-4ad013ae0dfc\") " pod="openshift-ingress-canary/ingress-canary-7swx2" Jul 06 20:35:42.958546 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.958522 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-installation-pull-secrets\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.958648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.958522 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-image-registry-private-configuration\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.963421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.963398 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8l2w\" (UniqueName: \"kubernetes.io/projected/f7c05db2-dcf9-406f-a781-4ad013ae0dfc-kube-api-access-t8l2w\") pod \"ingress-canary-7swx2\" (UID: \"f7c05db2-dcf9-406f-a781-4ad013ae0dfc\") " pod="openshift-ingress-canary/ingress-canary-7swx2" Jul 06 20:35:42.963421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.963409 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-bound-sa-token\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:42.966337 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.966298 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-776px\" (UniqueName: \"kubernetes.io/projected/18749804-aff5-4f10-a336-4562c73aff48-kube-api-access-776px\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:42.967630 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:42.967612 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdzbh\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-kube-api-access-xdzbh\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:43.066820 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.066785 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7swx2" Jul 06 20:35:43.453609 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.453577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:43.453774 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.453636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:43.453774 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:43.453737 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:35:43.453774 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:43.453751 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5867df54f9-rhdnn: secret "image-registry-tls" not found Jul 06 20:35:43.453923 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:43.453803 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls podName:3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:44.45378886 +0000 UTC m=+35.584151431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls") pod "image-registry-5867df54f9-rhdnn" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3") : secret "image-registry-tls" not found Jul 06 20:35:43.453923 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:43.453818 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:35:43.453923 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:43.453883 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls podName:18749804-aff5-4f10-a336-4562c73aff48 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:44.453863961 +0000 UTC m=+35.584226552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls") pod "dns-default-h7rh2" (UID: "18749804-aff5-4f10-a336-4562c73aff48") : secret "dns-default-metrics-tls" not found Jul 06 20:35:43.579757 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.579205 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:35:43.579757 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.579217 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:35:43.579988 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.579217 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:35:43.587146 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.586977 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 20:35:43.587146 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.587124 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 20:35:43.587679 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.587655 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 20:35:43.587679 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.587665 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 20:35:43.587679 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.587655 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c9msn\"" Jul 06 20:35:43.587912 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.587659 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sxc6n\"" Jul 06 20:35:43.751114 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.751043 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7swx2"] Jul 06 20:35:43.754601 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:35:43.754571 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c05db2_dcf9_406f_a781_4ad013ae0dfc.slice/crio-c16c6e27b0d63f57497abf0bf50afe7eca8971bde29f817878a6b7de72f046e4 WatchSource:0}: Error finding container c16c6e27b0d63f57497abf0bf50afe7eca8971bde29f817878a6b7de72f046e4: Status 404 returned error can't find the container with id c16c6e27b0d63f57497abf0bf50afe7eca8971bde29f817878a6b7de72f046e4 Jul 06 20:35:43.780393 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.780197 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7swx2" event={"ID":"f7c05db2-dcf9-406f-a781-4ad013ae0dfc","Type":"ContainerStarted","Data":"c16c6e27b0d63f57497abf0bf50afe7eca8971bde29f817878a6b7de72f046e4"} Jul 06 20:35:43.783095 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.783069 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9frbs" event={"ID":"a989d7ea-35b8-47ea-8942-a5af5d9718ab","Type":"ContainerStarted","Data":"49db9d1f93d3c0e3638db35684141b98f58fc85032a8180900fd0098f754993a"} Jul 06 20:35:43.989395 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:43.989320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqs8k_8bcc5ed5-4a67-416b-bebf-93316f005012/konnectivity-agent/0.log" limit=1024 Jul 06 20:35:44.462620 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:44.462591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:44.462620 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:44.462636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:44.462852 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:44.462744 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:35:44.462852 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:44.462762 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5867df54f9-rhdnn: secret "image-registry-tls" not found Jul 06 20:35:44.462852 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:44.462762 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:35:44.462852 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:44.462825 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls podName:18749804-aff5-4f10-a336-4562c73aff48 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:46.462807099 +0000 UTC m=+37.593169676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls") pod "dns-default-h7rh2" (UID: "18749804-aff5-4f10-a336-4562c73aff48") : secret "dns-default-metrics-tls" not found Jul 06 20:35:44.462852 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:44.462844 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls podName:3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:46.462835459 +0000 UTC m=+37.593198036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls") pod "image-registry-5867df54f9-rhdnn" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3") : secret "image-registry-tls" not found Jul 06 20:35:44.789088 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:44.788947 2579 generic.go:356] "Generic (PLEG): container finished" podID="a989d7ea-35b8-47ea-8942-a5af5d9718ab" containerID="49db9d1f93d3c0e3638db35684141b98f58fc85032a8180900fd0098f754993a" exitCode=0 Jul 06 20:35:44.789088 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:44.788992 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9frbs" event={"ID":"a989d7ea-35b8-47ea-8942-a5af5d9718ab","Type":"ContainerDied","Data":"49db9d1f93d3c0e3638db35684141b98f58fc85032a8180900fd0098f754993a"} Jul 06 20:35:45.794774 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:45.794741 2579 generic.go:356] "Generic (PLEG): container finished" podID="a989d7ea-35b8-47ea-8942-a5af5d9718ab" containerID="ee228e134cd284dc32b066a6e5a9b76901be89d350d6e5710e059edb88bb8839" exitCode=0 Jul 06 20:35:45.795191 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:45.794787 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9frbs" event={"ID":"a989d7ea-35b8-47ea-8942-a5af5d9718ab","Type":"ContainerDied","Data":"ee228e134cd284dc32b066a6e5a9b76901be89d350d6e5710e059edb88bb8839"} Jul 06 20:35:46.479355 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:46.479312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:46.479512 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:46.479400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:46.479512 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:46.479457 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:35:46.479512 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:46.479480 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5867df54f9-rhdnn: secret "image-registry-tls" not found Jul 06 20:35:46.479608 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:46.479533 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls podName:3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:50.479517258 +0000 UTC m=+41.609879829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls") pod "image-registry-5867df54f9-rhdnn" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3") : secret "image-registry-tls" not found Jul 06 20:35:46.479608 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:46.479536 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:35:46.479608 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:46.479591 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls podName:18749804-aff5-4f10-a336-4562c73aff48 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:50.479575444 +0000 UTC m=+41.609938032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls") pod "dns-default-h7rh2" (UID: "18749804-aff5-4f10-a336-4562c73aff48") : secret "dns-default-metrics-tls" not found Jul 06 20:35:46.798876 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:46.798797 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7swx2" event={"ID":"f7c05db2-dcf9-406f-a781-4ad013ae0dfc","Type":"ContainerStarted","Data":"b20b14843068dbff558deee532bfb130e19a9bbe03a050cf6c8ff654d3acfc6a"} Jul 06 20:35:46.801844 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:46.801819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9frbs" event={"ID":"a989d7ea-35b8-47ea-8942-a5af5d9718ab","Type":"ContainerStarted","Data":"e9ec16da49cfdf7ed1eb126168096f08e8cbe8563b45303a0acfae245ad86927"} Jul 06 20:35:46.817087 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:46.817036 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7swx2" podStartSLOduration=2.743587234 podStartE2EDuration="4.817019856s" totalImagesPullingTime="2.073432622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:46.816502583 +0000 UTC m=+37.946865178" watchObservedRunningTime="2026-07-06 20:35:46.817019856 +0000 UTC m=+37.947382448" Jul 06 20:35:46.844648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:46.844599 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9frbs" podStartSLOduration=9.149487776 podStartE2EDuration="37.844581524s" totalImagesPullingTime="28.695093748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:46.843594686 +0000 UTC m=+37.973957280" watchObservedRunningTime="2026-07-06 20:35:46.844581524 +0000 UTC m=+37.974944123" Jul 06 20:35:46.972606 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:46.972573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqs8k_8bcc5ed5-4a67-416b-bebf-93316f005012/konnectivity-agent/0.log" limit=1024 Jul 06 20:35:47.073874 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:47.073805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqs8k_8bcc5ed5-4a67-416b-bebf-93316f005012/konnectivity-agent/0.log" limit=1024 Jul 06 20:35:47.972211 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:47.972131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqs8k_8bcc5ed5-4a67-416b-bebf-93316f005012/konnectivity-agent/0.log" limit=1024 Jul 06 20:35:48.043621 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:48.043597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqs8k_8bcc5ed5-4a67-416b-bebf-93316f005012/konnectivity-agent/0.log" limit=1024 Jul 06 20:35:48.198384 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:48.198346 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqs8k_8bcc5ed5-4a67-416b-bebf-93316f005012/konnectivity-agent/0.log" limit=1024 Jul 06 20:35:48.298331 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:48.298263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqs8k_8bcc5ed5-4a67-416b-bebf-93316f005012/konnectivity-agent/0.log" limit=1024 Jul 06 20:35:50.510738 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:50.510699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:50.511156 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:50.510867 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:35:50.511156 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:50.510873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:50.511156 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:50.510938 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls podName:18749804-aff5-4f10-a336-4562c73aff48 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:58.51092173 +0000 UTC m=+49.641284302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls") pod "dns-default-h7rh2" (UID: "18749804-aff5-4f10-a336-4562c73aff48") : secret "dns-default-metrics-tls" not found Jul 06 20:35:50.511156 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:50.510943 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:35:50.511156 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:50.510980 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5867df54f9-rhdnn: secret "image-registry-tls" not found Jul 06 20:35:50.511156 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:50.511026 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls podName:3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:58.511012084 +0000 UTC m=+49.641374659 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls") pod "image-registry-5867df54f9-rhdnn" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3") : secret "image-registry-tls" not found Jul 06 20:35:58.564768 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:58.564734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:35:58.565176 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:35:58.564785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:35:58.565176 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:58.564875 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:35:58.565176 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:58.564879 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:35:58.565176 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:58.564973 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls podName:18749804-aff5-4f10-a336-4562c73aff48 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:14.564937316 +0000 UTC m=+65.695299888 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls") pod "dns-default-h7rh2" (UID: "18749804-aff5-4f10-a336-4562c73aff48") : secret "dns-default-metrics-tls" not found Jul 06 20:35:58.565176 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:58.564978 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5867df54f9-rhdnn: secret "image-registry-tls" not found Jul 06 20:35:58.565176 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:35:58.565018 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls podName:3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:14.565007304 +0000 UTC m=+65.695369889 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls") pod "image-registry-5867df54f9-rhdnn" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3") : secret "image-registry-tls" not found Jul 06 20:36:09.779766 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:09.779733 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqkjl" Jul 06 20:36:14.072222 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.072186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:36:14.075440 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.075422 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 20:36:14.086159 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.086131 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fb9f72c-98b1-4441-9c7d-57263e220a21-original-pull-secret\") pod \"global-pull-secret-syncer-kfgw5\" (UID: \"2fb9f72c-98b1-4441-9c7d-57263e220a21\") " pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:36:14.173528 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.173490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:36:14.176596 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.176578 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 20:36:14.184384 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:14.184366 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 20:36:14.184447 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:14.184436 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs podName:47c83a08-f54e-44ff-aaf6-cd03655c8b1e nodeName:}" failed. No retries permitted until 2026-07-06 20:37:18.184419228 +0000 UTC m=+129.314781800 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs") pod "network-metrics-daemon-c5bs6" (UID: "47c83a08-f54e-44ff-aaf6-cd03655c8b1e") : secret "metrics-daemon-secret" not found Jul 06 20:36:14.192813 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.192793 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kfgw5" Jul 06 20:36:14.274112 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.274069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jm5\" (UniqueName: \"kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5\") pod \"network-check-target-m6hmf\" (UID: \"76348c50-5f18-4580-95e2-9a92aba127dc\") " pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:36:14.276899 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.276859 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 20:36:14.289605 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.289583 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 20:36:14.297506 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.297479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jm5\" (UniqueName: \"kubernetes.io/projected/76348c50-5f18-4580-95e2-9a92aba127dc-kube-api-access-k4jm5\") pod \"network-check-target-m6hmf\" (UID: \"76348c50-5f18-4580-95e2-9a92aba127dc\") " pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:36:14.349720 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.349691 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kfgw5"] Jul 06 20:36:14.353291 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:36:14.353260 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb9f72c_98b1_4441_9c7d_57263e220a21.slice/crio-d893ad561322c44725b555d9b54eab54c6e92a8c379e68f53ad1acb795aae169 WatchSource:0}: Error finding container d893ad561322c44725b555d9b54eab54c6e92a8c379e68f53ad1acb795aae169: Status 404 returned error can't find the container with id d893ad561322c44725b555d9b54eab54c6e92a8c379e68f53ad1acb795aae169 Jul 06 20:36:14.502659 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.502629 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c9msn\"" Jul 06 20:36:14.510546 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.510519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:36:14.576636 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.576600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:36:14.576796 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:14.576743 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:36:14.576796 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:14.576762 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5867df54f9-rhdnn: secret "image-registry-tls" not found Jul 06 20:36:14.576796 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.576759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:36:14.576974 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:14.576808 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls podName:3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:46.57679553 +0000 UTC m=+97.707158101 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls") pod "image-registry-5867df54f9-rhdnn" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3") : secret "image-registry-tls" not found Jul 06 20:36:14.576974 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:14.576879 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:36:14.576974 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:14.576939 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls podName:18749804-aff5-4f10-a336-4562c73aff48 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:46.576921274 +0000 UTC m=+97.707283868 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls") pod "dns-default-h7rh2" (UID: "18749804-aff5-4f10-a336-4562c73aff48") : secret "dns-default-metrics-tls" not found Jul 06 20:36:14.658173 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.658136 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m6hmf"] Jul 06 20:36:14.660645 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:36:14.660616 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76348c50_5f18_4580_95e2_9a92aba127dc.slice/crio-0d3a31a9b5eb1c3917214e2d9e5da1e9928532cebfc6e8ee8da2e552433bd9b4 WatchSource:0}: Error finding container 0d3a31a9b5eb1c3917214e2d9e5da1e9928532cebfc6e8ee8da2e552433bd9b4: Status 404 returned error can't find the container with id 0d3a31a9b5eb1c3917214e2d9e5da1e9928532cebfc6e8ee8da2e552433bd9b4 Jul 06 20:36:14.866746 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.866657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m6hmf" event={"ID":"76348c50-5f18-4580-95e2-9a92aba127dc","Type":"ContainerStarted","Data":"0d3a31a9b5eb1c3917214e2d9e5da1e9928532cebfc6e8ee8da2e552433bd9b4"} Jul 06 20:36:14.867906 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:14.867876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kfgw5" event={"ID":"2fb9f72c-98b1-4441-9c7d-57263e220a21","Type":"ContainerStarted","Data":"d893ad561322c44725b555d9b54eab54c6e92a8c379e68f53ad1acb795aae169"} Jul 06 20:36:18.880739 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:18.880704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m6hmf" event={"ID":"76348c50-5f18-4580-95e2-9a92aba127dc","Type":"ContainerStarted","Data":"03ff747a07468ab3f8d84615d150f0cfc281b241fd9d3a6f102cc0219509c504"} Jul 06 20:36:18.881081 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:18.880865 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:36:18.902662 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:18.902619 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m6hmf" podStartSLOduration=65.745954997 podStartE2EDuration="1m9.902604578s" totalImagesPullingTime="4.156649581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:18.90207276 +0000 UTC m=+70.032435353" watchObservedRunningTime="2026-07-06 20:36:18.902604578 +0000 UTC m=+70.032967168" Jul 06 20:36:19.884725 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:19.884691 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kfgw5" event={"ID":"2fb9f72c-98b1-4441-9c7d-57263e220a21","Type":"ContainerStarted","Data":"a5d3d156aafe20bd65577a5f9f75867ce520910c4e703de9d7a5da56b70ceee5"} Jul 06 20:36:24.245621 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.245562 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kfgw5" podStartSLOduration=70.777959907 podStartE2EDuration="1m15.245545947s" totalImagesPullingTime="4.46758604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:19.901020663 +0000 UTC m=+71.031383257" watchObservedRunningTime="2026-07-06 20:36:24.245545947 +0000 UTC m=+75.375908546" Jul 06 20:36:24.246006 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.245871 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-77449d57c-gl4mt"] Jul 06 20:36:24.247665 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.247648 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.251225 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.251184 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 06 20:36:24.251354 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.251289 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8mrjc\"" Jul 06 20:36:24.252099 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.252081 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 06 20:36:24.252205 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.252084 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 06 20:36:24.252205 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.252191 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 20:36:24.252704 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.252676 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 06 20:36:24.252704 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.252689 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 20:36:24.274549 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.274529 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77449d57c-gl4mt"] Jul 06 20:36:24.353441 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.353412 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.353441 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.353443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.353616 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.353464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sz7l\" (UniqueName: \"kubernetes.io/projected/b7652700-4299-4f45-b55b-a0c4e91b4328-kube-api-access-8sz7l\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.353616 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.353550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-stats-auth\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.353616 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.353596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-default-certificate\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.454647 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.454610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.454647 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.454646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.454855 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.454662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sz7l\" (UniqueName: \"kubernetes.io/projected/b7652700-4299-4f45-b55b-a0c4e91b4328-kube-api-access-8sz7l\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.454855 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.454717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-stats-auth\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.454855 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.454737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-default-certificate\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.454855 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:24.454773 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 20:36:24.454855 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:24.454788 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle podName:b7652700-4299-4f45-b55b-a0c4e91b4328 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:24.954770963 +0000 UTC m=+76.085133538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle") pod "router-default-77449d57c-gl4mt" (UID: "b7652700-4299-4f45-b55b-a0c4e91b4328") : configmap references non-existent config key: service-ca.crt Jul 06 20:36:24.454855 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:24.454823 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs podName:b7652700-4299-4f45-b55b-a0c4e91b4328 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:24.954807403 +0000 UTC m=+76.085169974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs") pod "router-default-77449d57c-gl4mt" (UID: "b7652700-4299-4f45-b55b-a0c4e91b4328") : secret "router-metrics-certs-default" not found Jul 06 20:36:24.457158 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.457132 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-default-certificate\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.457331 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.457308 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-stats-auth\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.474334 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.474310 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sz7l\" (UniqueName: \"kubernetes.io/projected/b7652700-4299-4f45-b55b-a0c4e91b4328-kube-api-access-8sz7l\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.957997 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.957932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.957997 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:24.957994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:24.958226 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:24.958122 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle podName:b7652700-4299-4f45-b55b-a0c4e91b4328 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:25.958104824 +0000 UTC m=+77.088467395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle") pod "router-default-77449d57c-gl4mt" (UID: "b7652700-4299-4f45-b55b-a0c4e91b4328") : configmap references non-existent config key: service-ca.crt Jul 06 20:36:24.958226 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:24.958128 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 20:36:24.958226 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:24.958162 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs podName:b7652700-4299-4f45-b55b-a0c4e91b4328 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:25.95815381 +0000 UTC m=+77.088516382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs") pod "router-default-77449d57c-gl4mt" (UID: "b7652700-4299-4f45-b55b-a0c4e91b4328") : secret "router-metrics-certs-default" not found Jul 06 20:36:25.966066 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:25.966031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:25.966066 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:25.966069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:25.966508 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:25.966207 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 20:36:25.966508 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:25.966216 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle podName:b7652700-4299-4f45-b55b-a0c4e91b4328 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:27.966196444 +0000 UTC m=+79.096559015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle") pod "router-default-77449d57c-gl4mt" (UID: "b7652700-4299-4f45-b55b-a0c4e91b4328") : configmap references non-existent config key: service-ca.crt Jul 06 20:36:25.966508 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:25.966257 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs podName:b7652700-4299-4f45-b55b-a0c4e91b4328 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:27.966243768 +0000 UTC m=+79.096606339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs") pod "router-default-77449d57c-gl4mt" (UID: "b7652700-4299-4f45-b55b-a0c4e91b4328") : secret "router-metrics-certs-default" not found Jul 06 20:36:27.982938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:27.982892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:27.982938 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:27.982936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:27.983444 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:27.983067 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle podName:b7652700-4299-4f45-b55b-a0c4e91b4328 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:31.983048603 +0000 UTC m=+83.113411174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle") pod "router-default-77449d57c-gl4mt" (UID: "b7652700-4299-4f45-b55b-a0c4e91b4328") : configmap references non-existent config key: service-ca.crt Jul 06 20:36:27.983444 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:27.983108 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 20:36:27.983444 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:27.983161 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs podName:b7652700-4299-4f45-b55b-a0c4e91b4328 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:31.983149868 +0000 UTC m=+83.113512439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs") pod "router-default-77449d57c-gl4mt" (UID: "b7652700-4299-4f45-b55b-a0c4e91b4328") : secret "router-metrics-certs-default" not found Jul 06 20:36:30.114866 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.114834 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8h2x2"] Jul 06 20:36:30.116806 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.116791 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.118981 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.118942 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 20:36:30.119094 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.119043 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lh658\"" Jul 06 20:36:30.119166 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.119154 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 20:36:30.119594 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.119577 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 20:36:30.119672 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.119578 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 20:36:30.134625 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.134603 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8h2x2"] Jul 06 20:36:30.197662 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.197626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.197850 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.197683 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96e75ea1-1a37-45d1-ba33-92676762c7ce-crio-socket\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.197850 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.197825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.197978 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.197857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vl7q\" (UniqueName: \"kubernetes.io/projected/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-api-access-2vl7q\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.197978 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.197899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96e75ea1-1a37-45d1-ba33-92676762c7ce-data-volume\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.298572 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.298537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.298755 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.298602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96e75ea1-1a37-45d1-ba33-92676762c7ce-crio-socket\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.298755 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:30.298690 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:36:30.298755 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:30.298746 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:36:30.298926 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.298740 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96e75ea1-1a37-45d1-ba33-92676762c7ce-crio-socket\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.298926 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.298691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.298926 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:30.298751 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls podName:96e75ea1-1a37-45d1-ba33-92676762c7ce nodeName:}" failed. No retries permitted until 2026-07-06 20:36:30.798735009 +0000 UTC m=+81.929097581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8h2x2" (UID: "96e75ea1-1a37-45d1-ba33-92676762c7ce") : secret "insights-runtime-extractor-tls" not found Jul 06 20:36:30.298926 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:30.298829 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm podName:96e75ea1-1a37-45d1-ba33-92676762c7ce nodeName:}" failed. No retries permitted until 2026-07-06 20:36:30.798812722 +0000 UTC m=+81.929175299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8h2x2" (UID: "96e75ea1-1a37-45d1-ba33-92676762c7ce") : configmap "kube-rbac-proxy" not found Jul 06 20:36:30.298926 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.298859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vl7q\" (UniqueName: \"kubernetes.io/projected/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-api-access-2vl7q\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.298926 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.298896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96e75ea1-1a37-45d1-ba33-92676762c7ce-data-volume\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.299324 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.299305 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96e75ea1-1a37-45d1-ba33-92676762c7ce-data-volume\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.309916 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.309898 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vl7q\" (UniqueName: \"kubernetes.io/projected/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-api-access-2vl7q\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.802886 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.802851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.803062 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:30.802899 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:30.803062 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:30.803040 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:36:30.803130 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:30.803091 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:36:30.803162 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:30.803104 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls podName:96e75ea1-1a37-45d1-ba33-92676762c7ce nodeName:}" failed. No retries permitted until 2026-07-06 20:36:31.803090167 +0000 UTC m=+82.933452739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8h2x2" (UID: "96e75ea1-1a37-45d1-ba33-92676762c7ce") : secret "insights-runtime-extractor-tls" not found Jul 06 20:36:30.803204 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:30.803162 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm podName:96e75ea1-1a37-45d1-ba33-92676762c7ce nodeName:}" failed. No retries permitted until 2026-07-06 20:36:31.803144852 +0000 UTC m=+82.933507426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8h2x2" (UID: "96e75ea1-1a37-45d1-ba33-92676762c7ce") : configmap "kube-rbac-proxy" not found Jul 06 20:36:31.811014 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:31.810977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:31.811381 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:31.811140 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:36:31.811381 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:31.811180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:31.811381 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:31.811212 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls podName:96e75ea1-1a37-45d1-ba33-92676762c7ce nodeName:}" failed. No retries permitted until 2026-07-06 20:36:33.811195313 +0000 UTC m=+84.941557890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8h2x2" (UID: "96e75ea1-1a37-45d1-ba33-92676762c7ce") : secret "insights-runtime-extractor-tls" not found Jul 06 20:36:31.811381 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:31.811232 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:36:31.811381 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:31.811267 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm podName:96e75ea1-1a37-45d1-ba33-92676762c7ce nodeName:}" failed. No retries permitted until 2026-07-06 20:36:33.81125717 +0000 UTC m=+84.941619761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8h2x2" (UID: "96e75ea1-1a37-45d1-ba33-92676762c7ce") : configmap "kube-rbac-proxy" not found Jul 06 20:36:32.012646 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:32.012535 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:32.012646 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:32.012569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:32.012822 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:32.012664 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 20:36:32.012822 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:32.012724 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle podName:b7652700-4299-4f45-b55b-a0c4e91b4328 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:40.012697559 +0000 UTC m=+91.143060132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle") pod "router-default-77449d57c-gl4mt" (UID: "b7652700-4299-4f45-b55b-a0c4e91b4328") : configmap references non-existent config key: service-ca.crt Jul 06 20:36:32.012822 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:32.012746 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs podName:b7652700-4299-4f45-b55b-a0c4e91b4328 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:40.012739426 +0000 UTC m=+91.143101997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs") pod "router-default-77449d57c-gl4mt" (UID: "b7652700-4299-4f45-b55b-a0c4e91b4328") : secret "router-metrics-certs-default" not found Jul 06 20:36:32.107926 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:32.107897 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nsdj2_157e3ec8-eb18-430f-b4b5-397ab0281e2a/dns-node-resolver/0.log" Jul 06 20:36:33.113575 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:33.113545 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-whgpf_6f0d0297-cff0-4b11-b42c-af7c14b392a4/node-ca/0.log" Jul 06 20:36:33.708535 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:33.708496 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7swx2_f7c05db2-dcf9-406f-a781-4ad013ae0dfc/serve-healthcheck-canary/0.log" Jul 06 20:36:33.825012 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:33.824938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:33.825173 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:33.825045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:33.825173 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:33.825057 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:36:33.825173 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:33.825115 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm podName:96e75ea1-1a37-45d1-ba33-92676762c7ce nodeName:}" failed. No retries permitted until 2026-07-06 20:36:37.825101148 +0000 UTC m=+88.955463719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8h2x2" (UID: "96e75ea1-1a37-45d1-ba33-92676762c7ce") : configmap "kube-rbac-proxy" not found Jul 06 20:36:33.825173 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:33.825169 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:36:33.825312 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:33.825217 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls podName:96e75ea1-1a37-45d1-ba33-92676762c7ce nodeName:}" failed. No retries permitted until 2026-07-06 20:36:37.825207255 +0000 UTC m=+88.955569827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8h2x2" (UID: "96e75ea1-1a37-45d1-ba33-92676762c7ce") : secret "insights-runtime-extractor-tls" not found Jul 06 20:36:37.854268 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:37.854204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:37.854671 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:37.854300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:37.854671 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:37.854330 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:36:37.854671 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:37.854399 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm podName:96e75ea1-1a37-45d1-ba33-92676762c7ce nodeName:}" failed. No retries permitted until 2026-07-06 20:36:45.854382176 +0000 UTC m=+96.984744754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8h2x2" (UID: "96e75ea1-1a37-45d1-ba33-92676762c7ce") : configmap "kube-rbac-proxy" not found Jul 06 20:36:37.854671 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:37.854420 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:36:37.854671 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:37.854457 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls podName:96e75ea1-1a37-45d1-ba33-92676762c7ce nodeName:}" failed. No retries permitted until 2026-07-06 20:36:45.854446424 +0000 UTC m=+96.984808998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8h2x2" (UID: "96e75ea1-1a37-45d1-ba33-92676762c7ce") : secret "insights-runtime-extractor-tls" not found Jul 06 20:36:40.070376 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:40.070320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:40.070376 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:40.070377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:40.070884 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:36:40.070525 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle podName:b7652700-4299-4f45-b55b-a0c4e91b4328 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:56.070504404 +0000 UTC m=+107.200866986 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle") pod "router-default-77449d57c-gl4mt" (UID: "b7652700-4299-4f45-b55b-a0c4e91b4328") : configmap references non-existent config key: service-ca.crt Jul 06 20:36:40.072690 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:40.072672 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7652700-4299-4f45-b55b-a0c4e91b4328-metrics-certs\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:45.912593 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:45.912557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:45.913118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:45.912678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:45.913294 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:45.913273 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e75ea1-1a37-45d1-ba33-92676762c7ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:45.914871 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:45.914846 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e75ea1-1a37-45d1-ba33-92676762c7ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8h2x2\" (UID: \"96e75ea1-1a37-45d1-ba33-92676762c7ce\") " pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:46.026012 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.025946 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8h2x2" Jul 06 20:36:46.149269 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.149238 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8h2x2"] Jul 06 20:36:46.152503 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:36:46.152471 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e75ea1_1a37_45d1_ba33_92676762c7ce.slice/crio-1613b736bb643258944be2f712d6e7880b09f51899e83ae832d4c714ca3e873f WatchSource:0}: Error finding container 1613b736bb643258944be2f712d6e7880b09f51899e83ae832d4c714ca3e873f: Status 404 returned error can't find the container with id 1613b736bb643258944be2f712d6e7880b09f51899e83ae832d4c714ca3e873f Jul 06 20:36:46.619073 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.619035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:36:46.619235 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.619212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:36:46.621255 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.621232 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18749804-aff5-4f10-a336-4562c73aff48-metrics-tls\") pod \"dns-default-h7rh2\" (UID: \"18749804-aff5-4f10-a336-4562c73aff48\") " pod="openshift-dns/dns-default-h7rh2" Jul 06 20:36:46.621255 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.621246 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls\") pod \"image-registry-5867df54f9-rhdnn\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:36:46.657896 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.657575 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lqq25\"" Jul 06 20:36:46.665734 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.665703 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:36:46.678613 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.678589 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67jtg\"" Jul 06 20:36:46.686694 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.686672 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h7rh2" Jul 06 20:36:46.848410 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.848377 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5867df54f9-rhdnn"] Jul 06 20:36:46.852655 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:36:46.852625 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1ab6c3_3fc7_43c5_836a_bf4e20d112e3.slice/crio-aa35af4ccd7a1703ae735d86fe4e1de399d6fc90fa15c751298aac2292e8fb0e WatchSource:0}: Error finding container aa35af4ccd7a1703ae735d86fe4e1de399d6fc90fa15c751298aac2292e8fb0e: Status 404 returned error can't find the container with id aa35af4ccd7a1703ae735d86fe4e1de399d6fc90fa15c751298aac2292e8fb0e Jul 06 20:36:46.875694 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.875519 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h7rh2"] Jul 06 20:36:46.887524 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:36:46.887486 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18749804_aff5_4f10_a336_4562c73aff48.slice/crio-17aed56cb7199389bb6f7191526f387a9474de7dc55aa59985eaf9d4bfb71b19 WatchSource:0}: Error finding container 17aed56cb7199389bb6f7191526f387a9474de7dc55aa59985eaf9d4bfb71b19: Status 404 returned error can't find the container with id 17aed56cb7199389bb6f7191526f387a9474de7dc55aa59985eaf9d4bfb71b19 Jul 06 20:36:46.951393 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.951355 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8h2x2" event={"ID":"96e75ea1-1a37-45d1-ba33-92676762c7ce","Type":"ContainerStarted","Data":"1613b736bb643258944be2f712d6e7880b09f51899e83ae832d4c714ca3e873f"} Jul 06 20:36:46.953306 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.953215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h7rh2" event={"ID":"18749804-aff5-4f10-a336-4562c73aff48","Type":"ContainerStarted","Data":"17aed56cb7199389bb6f7191526f387a9474de7dc55aa59985eaf9d4bfb71b19"} Jul 06 20:36:46.955778 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.955599 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" event={"ID":"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3","Type":"ContainerStarted","Data":"128f07434580edcf68842ac19dd9c50c4d2ef6f8f7b2e90b9036e4c0ad455bf5"} Jul 06 20:36:46.955778 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.955662 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" event={"ID":"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3","Type":"ContainerStarted","Data":"aa35af4ccd7a1703ae735d86fe4e1de399d6fc90fa15c751298aac2292e8fb0e"} Jul 06 20:36:46.956454 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.956368 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:36:46.988620 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:46.988558 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" podStartSLOduration=96.988534901 podStartE2EDuration="1m36.988534901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:46.987459106 +0000 UTC m=+98.117821699" watchObservedRunningTime="2026-07-06 20:36:46.988534901 +0000 UTC m=+98.118897494" Jul 06 20:36:49.887798 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:49.887771 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m6hmf" Jul 06 20:36:51.976452 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:51.976411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8h2x2" event={"ID":"96e75ea1-1a37-45d1-ba33-92676762c7ce","Type":"ContainerStarted","Data":"03ac2983aad5c2e8c497cae01165d9c5fb8e36dddb3d57db8f799aa099965a5c"} Jul 06 20:36:51.978475 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:51.978443 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h7rh2" event={"ID":"18749804-aff5-4f10-a336-4562c73aff48","Type":"ContainerStarted","Data":"b0626e972fba10ca00f444f757967cc73bcbd45daefa01274741e9c7cfb51de5"} Jul 06 20:36:51.978607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:51.978501 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h7rh2" event={"ID":"18749804-aff5-4f10-a336-4562c73aff48","Type":"ContainerStarted","Data":"c03e38fc2f1a16f34c1e8b32ef5c27dffceeb91753f9e4399070ec38039d92f0"} Jul 06 20:36:51.978668 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:51.978613 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h7rh2" Jul 06 20:36:52.000818 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:52.000769 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h7rh2" podStartSLOduration=65.801270823 podStartE2EDuration="1m10.000753963s" totalImagesPullingTime="4.19948314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:51.999341489 +0000 UTC m=+103.129704092" watchObservedRunningTime="2026-07-06 20:36:52.000753963 +0000 UTC m=+103.131116565" Jul 06 20:36:52.984271 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:52.984228 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8h2x2" event={"ID":"96e75ea1-1a37-45d1-ba33-92676762c7ce","Type":"ContainerStarted","Data":"22efce77e4f3e517cce8a4da1973ac3c77dd6611b266d25416e68ccf12f02fa2"} Jul 06 20:36:54.502367 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.502342 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6"] Jul 06 20:36:54.504166 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.504147 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.506826 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.506811 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 20:36:54.507206 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.507188 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 20:36:54.507206 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.507201 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 20:36:54.507333 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.507189 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 20:36:54.507994 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.507978 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 20:36:54.508102 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.508086 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 20:36:54.508301 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.508287 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 20:36:54.535166 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.535144 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6"] Jul 06 20:36:54.584677 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.584658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7455009-b540-4a29-88d5-2f062f44a88c-hub\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.584777 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.584700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdm8\" (UniqueName: \"kubernetes.io/projected/e7455009-b540-4a29-88d5-2f062f44a88c-kube-api-access-rxdm8\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.584777 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.584727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7455009-b540-4a29-88d5-2f062f44a88c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.584777 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.584763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7455009-b540-4a29-88d5-2f062f44a88c-ca\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.584903 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.584780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7455009-b540-4a29-88d5-2f062f44a88c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.584903 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.584809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7455009-b540-4a29-88d5-2f062f44a88c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.685839 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.685807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7455009-b540-4a29-88d5-2f062f44a88c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.686003 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.685844 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7455009-b540-4a29-88d5-2f062f44a88c-ca\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.686003 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.685863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7455009-b540-4a29-88d5-2f062f44a88c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.686003 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.685885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7455009-b540-4a29-88d5-2f062f44a88c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.686158 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.686013 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7455009-b540-4a29-88d5-2f062f44a88c-hub\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.686158 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.686083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdm8\" (UniqueName: \"kubernetes.io/projected/e7455009-b540-4a29-88d5-2f062f44a88c-kube-api-access-rxdm8\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.686632 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.686610 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7455009-b540-4a29-88d5-2f062f44a88c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.688471 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.688445 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7455009-b540-4a29-88d5-2f062f44a88c-hub\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.688471 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.688466 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7455009-b540-4a29-88d5-2f062f44a88c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.688708 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.688693 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7455009-b540-4a29-88d5-2f062f44a88c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.688749 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.688696 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7455009-b540-4a29-88d5-2f062f44a88c-ca\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.701044 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.701023 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdm8\" (UniqueName: \"kubernetes.io/projected/e7455009-b540-4a29-88d5-2f062f44a88c-kube-api-access-rxdm8\") pod \"cluster-proxy-proxy-agent-655b79495d-fxbv6\" (UID: \"e7455009-b540-4a29-88d5-2f062f44a88c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.840902 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.840832 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" Jul 06 20:36:54.961466 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.961429 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6"] Jul 06 20:36:54.993066 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.993038 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8h2x2" event={"ID":"96e75ea1-1a37-45d1-ba33-92676762c7ce","Type":"ContainerStarted","Data":"07ee1051b5ffbd7bdf40dfca8e428cbf0c87c42395cb9cb7ac042c1543f91ca7"} Jul 06 20:36:54.994169 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:54.994147 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" event={"ID":"e7455009-b540-4a29-88d5-2f062f44a88c","Type":"ContainerStarted","Data":"28cddf75769dc63960b06284e22ecc6bc4a121cc34e240a720c6748e4cffd9cd"} Jul 06 20:36:55.034321 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:55.034277 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8h2x2" podStartSLOduration=16.851915483 podStartE2EDuration="25.034263367s" totalImagesPullingTime="8.182347884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:36:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:55.033359983 +0000 UTC m=+106.163722586" watchObservedRunningTime="2026-07-06 20:36:55.034263367 +0000 UTC m=+106.164625959" Jul 06 20:36:56.095892 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:56.095538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:56.096386 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:56.096353 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7652700-4299-4f45-b55b-a0c4e91b4328-service-ca-bundle\") pod \"router-default-77449d57c-gl4mt\" (UID: \"b7652700-4299-4f45-b55b-a0c4e91b4328\") " pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:56.356437 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:56.356392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:57.243497 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:57.243466 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77449d57c-gl4mt"] Jul 06 20:36:58.294449 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:36:58.294409 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7652700_4299_4f45_b55b_a0c4e91b4328.slice/crio-d08723fe92431e00ca05d2ae7268eb2f27697128ffb14ad21ace1c586ed04d3d WatchSource:0}: Error finding container d08723fe92431e00ca05d2ae7268eb2f27697128ffb14ad21ace1c586ed04d3d: Status 404 returned error can't find the container with id d08723fe92431e00ca05d2ae7268eb2f27697128ffb14ad21ace1c586ed04d3d Jul 06 20:36:59.011147 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:59.011097 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" event={"ID":"e7455009-b540-4a29-88d5-2f062f44a88c","Type":"ContainerStarted","Data":"cc2332ae39941dba733206cc8d133d9969e5511b36c557dc23e9e84b0df4b6d8"} Jul 06 20:36:59.011328 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:59.011153 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" event={"ID":"e7455009-b540-4a29-88d5-2f062f44a88c","Type":"ContainerStarted","Data":"0e98655b7ec5cd108dc6b15a794644de3e41173dd374b4be39fa5f221a183e86"} Jul 06 20:36:59.011328 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:59.011171 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" event={"ID":"e7455009-b540-4a29-88d5-2f062f44a88c","Type":"ContainerStarted","Data":"7d1caf465edfebdcc4405f00d9e8749eb212595155efa11ae6cdadc9de9fb7c8"} Jul 06 20:36:59.012467 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:59.012442 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77449d57c-gl4mt" event={"ID":"b7652700-4299-4f45-b55b-a0c4e91b4328","Type":"ContainerStarted","Data":"c06ea1660e61cd0f546f6f916dfc4f29d9aedae541e952f8121c1a5be5bc8759"} Jul 06 20:36:59.012592 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:59.012525 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77449d57c-gl4mt" event={"ID":"b7652700-4299-4f45-b55b-a0c4e91b4328","Type":"ContainerStarted","Data":"d08723fe92431e00ca05d2ae7268eb2f27697128ffb14ad21ace1c586ed04d3d"} Jul 06 20:36:59.040654 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:59.040610 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655b79495d-fxbv6" podStartSLOduration=1.6420002550000001 podStartE2EDuration="5.040596905s" totalImagesPullingTime="3.39859665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:59.039644642 +0000 UTC m=+110.170007234" watchObservedRunningTime="2026-07-06 20:36:59.040596905 +0000 UTC m=+110.170959500" Jul 06 20:36:59.067227 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:59.067190 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-77449d57c-gl4mt" podStartSLOduration=35.067179442 podStartE2EDuration="35.067179442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:59.066055527 +0000 UTC m=+110.196418120" watchObservedRunningTime="2026-07-06 20:36:59.067179442 +0000 UTC m=+110.197542013" Jul 06 20:36:59.357590 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:59.357562 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:36:59.360236 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:36:59.360212 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:37:00.015993 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:00.015965 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:37:00.017071 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:00.017050 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-77449d57c-gl4mt" Jul 06 20:37:01.986518 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:01.986486 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h7rh2" Jul 06 20:37:08.141876 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.141847 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ltmvn"] Jul 06 20:37:08.145022 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.145000 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.147053 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.147028 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 20:37:08.147197 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.147180 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-57hfd\"" Jul 06 20:37:08.147256 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.147243 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 20:37:08.147317 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.147300 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 20:37:08.147378 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.147340 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 20:37:08.147728 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.147713 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 20:37:08.147790 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.147778 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 20:37:08.280978 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.280934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-textfile\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.280978 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.280989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/248602df-f8ef-438f-9a63-f9149313677b-sys\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.281184 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.281009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/248602df-f8ef-438f-9a63-f9149313677b-metrics-client-ca\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.281184 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.281072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hksdf\" (UniqueName: \"kubernetes.io/projected/248602df-f8ef-438f-9a63-f9149313677b-kube-api-access-hksdf\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.281184 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.281116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.281184 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.281141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-tls\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.281184 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.281183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-wtmp\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.281335 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.281198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.281335 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.281226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/248602df-f8ef-438f-9a63-f9149313677b-root\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382037 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksdf\" (UniqueName: \"kubernetes.io/projected/248602df-f8ef-438f-9a63-f9149313677b-kube-api-access-hksdf\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382207 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382207 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-tls\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382207 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-wtmp\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382207 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382207 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/248602df-f8ef-438f-9a63-f9149313677b-root\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382444 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382243 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/248602df-f8ef-438f-9a63-f9149313677b-root\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382444 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382319 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-wtmp\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382444 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-textfile\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382444 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/248602df-f8ef-438f-9a63-f9149313677b-sys\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382444 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/248602df-f8ef-438f-9a63-f9149313677b-metrics-client-ca\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382654 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382516 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/248602df-f8ef-438f-9a63-f9149313677b-sys\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382699 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382675 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-textfile\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.382777 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.382755 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.383096 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.383075 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/248602df-f8ef-438f-9a63-f9149313677b-metrics-client-ca\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.384552 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.384525 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-tls\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.384607 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.384553 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/248602df-f8ef-438f-9a63-f9149313677b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.401390 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.401334 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksdf\" (UniqueName: \"kubernetes.io/projected/248602df-f8ef-438f-9a63-f9149313677b-kube-api-access-hksdf\") pod \"node-exporter-ltmvn\" (UID: \"248602df-f8ef-438f-9a63-f9149313677b\") " pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.453514 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.453484 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ltmvn" Jul 06 20:37:08.464068 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:37:08.464045 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248602df_f8ef_438f_9a63_f9149313677b.slice/crio-35bdb7497e07cb333d7d897e6a5d59740a69c9483fd8b4712e1a30d3cb2f1f56 WatchSource:0}: Error finding container 35bdb7497e07cb333d7d897e6a5d59740a69c9483fd8b4712e1a30d3cb2f1f56: Status 404 returned error can't find the container with id 35bdb7497e07cb333d7d897e6a5d59740a69c9483fd8b4712e1a30d3cb2f1f56 Jul 06 20:37:08.969572 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:08.969544 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:37:09.049373 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.049333 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ltmvn" event={"ID":"248602df-f8ef-438f-9a63-f9149313677b","Type":"ContainerStarted","Data":"35bdb7497e07cb333d7d897e6a5d59740a69c9483fd8b4712e1a30d3cb2f1f56"} Jul 06 20:37:09.276207 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.276177 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 20:37:09.280036 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.280018 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.282140 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.282124 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 20:37:09.282752 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.282728 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 20:37:09.283112 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.283097 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dwwcc\"" Jul 06 20:37:09.283233 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.283214 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 20:37:09.283498 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.283483 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 20:37:09.283638 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.283621 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 20:37:09.284288 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.284273 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 20:37:09.286501 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.286393 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 20:37:09.286501 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.286416 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 20:37:09.297303 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.297284 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 20:37:09.308231 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.308211 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 20:37:09.389788 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.389753 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.389981 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.389814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a250e53-7b99-424b-acaa-c50ad6d8781e-config-out\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.389981 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.389852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m64tm\" (UniqueName: \"kubernetes.io/projected/2a250e53-7b99-424b-acaa-c50ad6d8781e-kube-api-access-m64tm\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.389981 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.389887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.389981 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.389923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.390195 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.390031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.390195 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.390078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-config-volume\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.390195 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.390109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a250e53-7b99-424b-acaa-c50ad6d8781e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.390195 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.390127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a250e53-7b99-424b-acaa-c50ad6d8781e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.390195 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.390159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a250e53-7b99-424b-acaa-c50ad6d8781e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.390372 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.390349 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-web-config\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.390411 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.390396 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a250e53-7b99-424b-acaa-c50ad6d8781e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.390463 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.390438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.491487 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.491449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a250e53-7b99-424b-acaa-c50ad6d8781e-config-out\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.491487 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.491488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m64tm\" (UniqueName: \"kubernetes.io/projected/2a250e53-7b99-424b-acaa-c50ad6d8781e-kube-api-access-m64tm\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.491697 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.491518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.491697 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.491551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.491697 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.491577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.491697 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.491602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-config-volume\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.491697 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.491636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a250e53-7b99-424b-acaa-c50ad6d8781e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.491697 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.491659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a250e53-7b99-424b-acaa-c50ad6d8781e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.491697 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.491681 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a250e53-7b99-424b-acaa-c50ad6d8781e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.492050 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.491712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-web-config\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.492050 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:37:09.491720 2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 06 20:37:09.492050 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.491746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a250e53-7b99-424b-acaa-c50ad6d8781e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.492050 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:37:09.491796 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-main-tls podName:2a250e53-7b99-424b-acaa-c50ad6d8781e nodeName:}" failed. No retries permitted until 2026-07-06 20:37:09.991775174 +0000 UTC m=+121.122137747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "2a250e53-7b99-424b-acaa-c50ad6d8781e") : secret "alertmanager-main-tls" not found Jul 06 20:37:09.492050 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.491822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.492050 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.491864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.492364 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.492144 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a250e53-7b99-424b-acaa-c50ad6d8781e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.492800 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.492768 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a250e53-7b99-424b-acaa-c50ad6d8781e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.493355 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.493310 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a250e53-7b99-424b-acaa-c50ad6d8781e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.494544 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.494520 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a250e53-7b99-424b-acaa-c50ad6d8781e-config-out\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.494705 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.494661 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.494971 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.494935 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-web-config\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.495017 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.494941 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a250e53-7b99-424b-acaa-c50ad6d8781e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.495591 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.495570 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.495727 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.495704 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.495810 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.495794 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-config-volume\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.496115 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.496095 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.506778 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.506757 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m64tm\" (UniqueName: \"kubernetes.io/projected/2a250e53-7b99-424b-acaa-c50ad6d8781e-kube-api-access-m64tm\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.995419 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.995383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:09.997735 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:09.997715 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a250e53-7b99-424b-acaa-c50ad6d8781e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2a250e53-7b99-424b-acaa-c50ad6d8781e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:10.054023 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:10.053989 2579 generic.go:356] "Generic (PLEG): container finished" podID="248602df-f8ef-438f-9a63-f9149313677b" containerID="c80a55725d26fc15b5c7c683930d8189ca861c4ca586d8138b391188fbe227eb" exitCode=0 Jul 06 20:37:10.054182 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:10.054042 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ltmvn" event={"ID":"248602df-f8ef-438f-9a63-f9149313677b","Type":"ContainerDied","Data":"c80a55725d26fc15b5c7c683930d8189ca861c4ca586d8138b391188fbe227eb"} Jul 06 20:37:10.190037 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:10.190012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 20:37:10.332494 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:10.332332 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 20:37:10.334840 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:37:10.334816 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a250e53_7b99_424b_acaa_c50ad6d8781e.slice/crio-a4afd96300650acaf0b528bb8af7a1608ae3137d263c324956026a9715660f30 WatchSource:0}: Error finding container a4afd96300650acaf0b528bb8af7a1608ae3137d263c324956026a9715660f30: Status 404 returned error can't find the container with id a4afd96300650acaf0b528bb8af7a1608ae3137d263c324956026a9715660f30 Jul 06 20:37:11.058842 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:11.058799 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a250e53-7b99-424b-acaa-c50ad6d8781e","Type":"ContainerStarted","Data":"a4afd96300650acaf0b528bb8af7a1608ae3137d263c324956026a9715660f30"} Jul 06 20:37:11.061007 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:11.060979 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ltmvn" event={"ID":"248602df-f8ef-438f-9a63-f9149313677b","Type":"ContainerStarted","Data":"b5a036dff3215ccab32170a278fba7ca59f5b04053421c6790475b0304df9240"} Jul 06 20:37:11.061121 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:11.061031 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ltmvn" event={"ID":"248602df-f8ef-438f-9a63-f9149313677b","Type":"ContainerStarted","Data":"2fb144e86cb17a313a3c2290b79143c0b0233c7ebced1134a5d72d64529861d6"} Jul 06 20:37:11.083905 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:11.083850 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ltmvn" podStartSLOduration=2.203553335 podStartE2EDuration="3.083836096s" totalImagesPullingTime="810.656989ms" totalInitContainerRuntime="69.625772ms" isStatefulPod=true podCreationTimestamp="2026-07-06 20:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:11.082424353 +0000 UTC m=+122.212786969" watchObservedRunningTime="2026-07-06 20:37:11.083836096 +0000 UTC m=+122.214198752" Jul 06 20:37:12.070100 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:12.070064 2579 generic.go:356] "Generic (PLEG): container finished" podID="2a250e53-7b99-424b-acaa-c50ad6d8781e" containerID="b2668484634da9ff8c9a670729058a0ff9fa50e90596408e4f13638998b51a2c" exitCode=0 Jul 06 20:37:12.070477 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:12.070142 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a250e53-7b99-424b-acaa-c50ad6d8781e","Type":"ContainerDied","Data":"b2668484634da9ff8c9a670729058a0ff9fa50e90596408e4f13638998b51a2c"} Jul 06 20:37:14.079484 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:14.079451 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a250e53-7b99-424b-acaa-c50ad6d8781e","Type":"ContainerStarted","Data":"e18e395493bb31fe800d5ed844e1f4b3bdd646786424be182b2d715f8ea4583b"} Jul 06 20:37:15.086182 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.086148 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a250e53-7b99-424b-acaa-c50ad6d8781e","Type":"ContainerStarted","Data":"6185f29f26f266a44936d9876f4df86b785a718e557f85316076ad8254270f1d"} Jul 06 20:37:15.086551 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.086225 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a250e53-7b99-424b-acaa-c50ad6d8781e","Type":"ContainerStarted","Data":"ab5efa0f7597439cef98ac0f7ad5b4ffffbc681aa185ac0aa4df195a30a1bbf9"} Jul 06 20:37:15.086551 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.086253 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a250e53-7b99-424b-acaa-c50ad6d8781e","Type":"ContainerStarted","Data":"898187728d98d629645f47cdf1769f25026a005f0fb0b1ec9ff6b4575a3bf2d4"} Jul 06 20:37:15.086551 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.086275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a250e53-7b99-424b-acaa-c50ad6d8781e","Type":"ContainerStarted","Data":"385fd05fb905409ae1a6a239d039b6088a9c70859a5a5efa916189dcf5278b44"} Jul 06 20:37:15.296354 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.296324 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f9476f465-mwwnq"] Jul 06 20:37:15.299370 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.299352 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.301331 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.301310 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 20:37:15.301633 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.301613 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 20:37:15.301878 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.301859 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 20:37:15.301878 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.301872 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 20:37:15.302057 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.301875 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qzqw5\"" Jul 06 20:37:15.302057 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.301899 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 20:37:15.302205 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.302188 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 20:37:15.302249 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.302213 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 20:37:15.307829 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.307813 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 20:37:15.318129 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.318071 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f9476f465-mwwnq"] Jul 06 20:37:15.443140 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.443108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-serving-cert\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.443140 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.443140 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-oauth-config\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.443359 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.443186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-config\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.443359 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.443240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-service-ca\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.443359 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.443273 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfnwv\" (UniqueName: \"kubernetes.io/projected/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-kube-api-access-dfnwv\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.443359 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.443331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-oauth-serving-cert\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.443489 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.443366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-trusted-ca-bundle\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.544636 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.544597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-serving-cert\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.544636 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.544635 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-oauth-config\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.544853 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.544787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-config\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.544853 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.544822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-service-ca\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.544853 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.544839 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfnwv\" (UniqueName: \"kubernetes.io/projected/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-kube-api-access-dfnwv\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.545049 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.544875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-oauth-serving-cert\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.545049 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.544897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-trusted-ca-bundle\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.545563 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.545536 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-config\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.545563 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.545560 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-service-ca\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.545755 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.545730 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-oauth-serving-cert\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.545795 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.545768 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-trusted-ca-bundle\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.547567 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.547541 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-oauth-config\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.547653 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.547632 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-serving-cert\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.555339 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.555318 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfnwv\" (UniqueName: \"kubernetes.io/projected/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-kube-api-access-dfnwv\") pod \"console-7f9476f465-mwwnq\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.608928 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.608894 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:15.737133 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:15.737099 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f9476f465-mwwnq"] Jul 06 20:37:15.740392 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:37:15.740366 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa3c7a7_0c9a_4107_9244_829fe6c9918b.slice/crio-f2c4e65686648abafee0adfa9b3647f2c645e681514269e3409a781e4feb33c3 WatchSource:0}: Error finding container f2c4e65686648abafee0adfa9b3647f2c645e681514269e3409a781e4feb33c3: Status 404 returned error can't find the container with id f2c4e65686648abafee0adfa9b3647f2c645e681514269e3409a781e4feb33c3 Jul 06 20:37:16.091026 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:16.090904 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9476f465-mwwnq" event={"ID":"9fa3c7a7-0c9a-4107-9244-829fe6c9918b","Type":"ContainerStarted","Data":"f2c4e65686648abafee0adfa9b3647f2c645e681514269e3409a781e4feb33c3"} Jul 06 20:37:16.093906 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:16.093878 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a250e53-7b99-424b-acaa-c50ad6d8781e","Type":"ContainerStarted","Data":"a20597b62a41d6910b2fbad0a12426285686c7754c7a89279681b6803063b750"} Jul 06 20:37:16.128896 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:16.128842 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.397793783 podStartE2EDuration="7.128825054s" totalImagesPullingTime="3.711960067s" totalInitContainerRuntime="19.071204ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:16.12804044 +0000 UTC m=+127.258403030" watchObservedRunningTime="2026-07-06 20:37:16.128825054 +0000 UTC m=+127.259187646" Jul 06 20:37:16.334691 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:16.334657 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5867df54f9-rhdnn"] Jul 06 20:37:18.270332 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:18.270285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:37:18.272966 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:18.272921 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c83a08-f54e-44ff-aaf6-cd03655c8b1e-metrics-certs\") pod \"network-metrics-daemon-c5bs6\" (UID: \"47c83a08-f54e-44ff-aaf6-cd03655c8b1e\") " pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:37:18.406832 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:18.406799 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sxc6n\"" Jul 06 20:37:18.414699 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:18.414656 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5bs6" Jul 06 20:37:18.948214 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:18.948182 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c5bs6"] Jul 06 20:37:18.951401 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:37:18.951374 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c83a08_f54e_44ff_aaf6_cd03655c8b1e.slice/crio-5712a9603005ee28cfb8d8f509e08b92273fcd14e7eb6633f5ae919a387611fc WatchSource:0}: Error finding container 5712a9603005ee28cfb8d8f509e08b92273fcd14e7eb6633f5ae919a387611fc: Status 404 returned error can't find the container with id 5712a9603005ee28cfb8d8f509e08b92273fcd14e7eb6633f5ae919a387611fc Jul 06 20:37:19.110245 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:19.110207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c5bs6" event={"ID":"47c83a08-f54e-44ff-aaf6-cd03655c8b1e","Type":"ContainerStarted","Data":"5712a9603005ee28cfb8d8f509e08b92273fcd14e7eb6633f5ae919a387611fc"} Jul 06 20:37:19.111673 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:19.111650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9476f465-mwwnq" event={"ID":"9fa3c7a7-0c9a-4107-9244-829fe6c9918b","Type":"ContainerStarted","Data":"88de0601cb9744686beec1bfd502bddb8c50d32480f770ed25a75cdb7e8bfec6"} Jul 06 20:37:19.132118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:19.132065 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f9476f465-mwwnq" podStartSLOduration=1.001730949 podStartE2EDuration="4.132047611s" totalImagesPullingTime="3.130316662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:19.130890265 +0000 UTC m=+130.261252857" watchObservedRunningTime="2026-07-06 20:37:19.132047611 +0000 UTC m=+130.262410202" Jul 06 20:37:20.117382 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:20.117345 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c5bs6" event={"ID":"47c83a08-f54e-44ff-aaf6-cd03655c8b1e","Type":"ContainerStarted","Data":"16251473b12bdd5df5b33455046238d85a79fb4e0a9e54afc5acf19d637ea40d"} Jul 06 20:37:20.117836 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:20.117403 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c5bs6" event={"ID":"47c83a08-f54e-44ff-aaf6-cd03655c8b1e","Type":"ContainerStarted","Data":"6a4fcc730f76dd0a77c97d8b63b85851acd557a0103cde7287acbfd890985298"} Jul 06 20:37:20.142692 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:20.142633 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c5bs6" podStartSLOduration=130.278525001 podStartE2EDuration="2m11.14261762s" totalImagesPullingTime="864.092619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:20.140692951 +0000 UTC m=+131.271055549" watchObservedRunningTime="2026-07-06 20:37:20.14261762 +0000 UTC m=+131.272980212" Jul 06 20:37:25.609662 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:25.609617 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:25.610149 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:25.609680 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:25.614456 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:25.614433 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:26.140992 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:26.140947 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:37:41.354392 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.354328 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" podUID="3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" containerName="registry" containerID="cri-o://128f07434580edcf68842ac19dd9c50c4d2ef6f8f7b2e90b9036e4c0ad455bf5" gracePeriod=30 Jul 06 20:37:41.604209 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.604181 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:37:41.639970 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.639868 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-installation-pull-secrets\") pod \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " Jul 06 20:37:41.639970 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.639908 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls\") pod \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " Jul 06 20:37:41.639970 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.639940 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-image-registry-private-configuration\") pod \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " Jul 06 20:37:41.640236 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.639988 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-trusted-ca\") pod \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " Jul 06 20:37:41.640236 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.640014 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-ca-trust-extracted\") pod \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " Jul 06 20:37:41.640236 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.640030 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-bound-sa-token\") pod \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " Jul 06 20:37:41.640236 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.640055 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdzbh\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-kube-api-access-xdzbh\") pod \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " Jul 06 20:37:41.640236 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.640083 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-certificates\") pod \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\" (UID: \"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3\") " Jul 06 20:37:41.640489 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.640459 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:37:41.640544 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.640527 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:37:41.642716 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.642677 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:37:41.642716 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.642670 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:37:41.642909 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.642760 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:37:41.642986 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.642928 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:37:41.643261 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.643237 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-kube-api-access-xdzbh" (OuterVolumeSpecName: "kube-api-access-xdzbh") pod "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3"). InnerVolumeSpecName "kube-api-access-xdzbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:37:41.649252 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.649224 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" (UID: "3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:37:41.741296 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.741256 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdzbh\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-kube-api-access-xdzbh\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:37:41.741296 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.741292 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-certificates\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:37:41.741296 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.741301 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-installation-pull-secrets\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:37:41.741505 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.741311 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-registry-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:37:41.741505 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.741321 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-image-registry-private-configuration\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:37:41.741505 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.741329 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-trusted-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:37:41.741505 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.741337 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-ca-trust-extracted\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:37:41.741505 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:41.741345 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3-bound-sa-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:37:42.196366 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:42.196327 2579 generic.go:356] "Generic (PLEG): container finished" podID="3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" containerID="128f07434580edcf68842ac19dd9c50c4d2ef6f8f7b2e90b9036e4c0ad455bf5" exitCode=0 Jul 06 20:37:42.196530 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:42.196403 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" Jul 06 20:37:42.196530 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:42.196397 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" event={"ID":"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3","Type":"ContainerDied","Data":"128f07434580edcf68842ac19dd9c50c4d2ef6f8f7b2e90b9036e4c0ad455bf5"} Jul 06 20:37:42.196530 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:42.196518 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5867df54f9-rhdnn" event={"ID":"3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3","Type":"ContainerDied","Data":"aa35af4ccd7a1703ae735d86fe4e1de399d6fc90fa15c751298aac2292e8fb0e"} Jul 06 20:37:42.196629 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:42.196543 2579 scope.go:117] "RemoveContainer" containerID="128f07434580edcf68842ac19dd9c50c4d2ef6f8f7b2e90b9036e4c0ad455bf5" Jul 06 20:37:42.204908 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:42.204885 2579 scope.go:117] "RemoveContainer" containerID="128f07434580edcf68842ac19dd9c50c4d2ef6f8f7b2e90b9036e4c0ad455bf5" Jul 06 20:37:42.205216 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:37:42.205196 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128f07434580edcf68842ac19dd9c50c4d2ef6f8f7b2e90b9036e4c0ad455bf5\": container with ID starting with 128f07434580edcf68842ac19dd9c50c4d2ef6f8f7b2e90b9036e4c0ad455bf5 not found: ID does not exist" containerID="128f07434580edcf68842ac19dd9c50c4d2ef6f8f7b2e90b9036e4c0ad455bf5" Jul 06 20:37:42.205279 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:42.205225 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128f07434580edcf68842ac19dd9c50c4d2ef6f8f7b2e90b9036e4c0ad455bf5"} err="failed to get container status \"128f07434580edcf68842ac19dd9c50c4d2ef6f8f7b2e90b9036e4c0ad455bf5\": rpc error: code = NotFound desc = could not find container \"128f07434580edcf68842ac19dd9c50c4d2ef6f8f7b2e90b9036e4c0ad455bf5\": container with ID starting with 128f07434580edcf68842ac19dd9c50c4d2ef6f8f7b2e90b9036e4c0ad455bf5 not found: ID does not exist" Jul 06 20:37:42.220140 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:42.220110 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5867df54f9-rhdnn"] Jul 06 20:37:42.226935 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:42.226910 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5867df54f9-rhdnn"] Jul 06 20:37:43.583515 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:37:43.583484 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" path="/var/lib/kubelet/pods/3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3/volumes" Jul 06 20:38:32.596996 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.596947 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-54876475cf-s9rxb"] Jul 06 20:38:32.597475 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.597245 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" containerName="registry" Jul 06 20:38:32.597475 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.597257 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" containerName="registry" Jul 06 20:38:32.597475 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.597298 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1ab6c3-3fc7-43c5-836a-bf4e20d112e3" containerName="registry" Jul 06 20:38:32.600160 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.600142 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.602292 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.602245 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 20:38:32.602439 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.602309 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 20:38:32.602439 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.602367 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wfbgh\"" Jul 06 20:38:32.602439 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.602393 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 20:38:32.602597 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.602583 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 20:38:32.602794 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.602780 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 20:38:32.608188 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.608164 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 20:38:32.614782 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.614758 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54876475cf-s9rxb"] Jul 06 20:38:32.722532 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.722493 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fb55fa3-ff25-4f04-8785-2c218ab4013d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.722739 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.722548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fb55fa3-ff25-4f04-8785-2c218ab4013d-serving-certs-ca-bundle\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.722739 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.722575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5fb55fa3-ff25-4f04-8785-2c218ab4013d-secret-telemeter-client\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.722739 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.722595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5fb55fa3-ff25-4f04-8785-2c218ab4013d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.722739 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.722614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5fb55fa3-ff25-4f04-8785-2c218ab4013d-federate-client-tls\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.722739 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.722638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fb55fa3-ff25-4f04-8785-2c218ab4013d-metrics-client-ca\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.723046 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.722794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5fb55fa3-ff25-4f04-8785-2c218ab4013d-telemeter-client-tls\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.723046 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.722914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqrrm\" (UniqueName: \"kubernetes.io/projected/5fb55fa3-ff25-4f04-8785-2c218ab4013d-kube-api-access-nqrrm\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.823994 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.823938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fb55fa3-ff25-4f04-8785-2c218ab4013d-serving-certs-ca-bundle\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.824157 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.824007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5fb55fa3-ff25-4f04-8785-2c218ab4013d-secret-telemeter-client\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.824157 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.824042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5fb55fa3-ff25-4f04-8785-2c218ab4013d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.824279 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.824078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5fb55fa3-ff25-4f04-8785-2c218ab4013d-federate-client-tls\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.824633 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.824473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fb55fa3-ff25-4f04-8785-2c218ab4013d-metrics-client-ca\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.825072 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.824841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5fb55fa3-ff25-4f04-8785-2c218ab4013d-telemeter-client-tls\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.825256 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.825228 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fb55fa3-ff25-4f04-8785-2c218ab4013d-metrics-client-ca\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.825423 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.825244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrrm\" (UniqueName: \"kubernetes.io/projected/5fb55fa3-ff25-4f04-8785-2c218ab4013d-kube-api-access-nqrrm\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.825599 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.825471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fb55fa3-ff25-4f04-8785-2c218ab4013d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.826244 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.825159 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fb55fa3-ff25-4f04-8785-2c218ab4013d-serving-certs-ca-bundle\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.830594 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.827568 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5fb55fa3-ff25-4f04-8785-2c218ab4013d-secret-telemeter-client\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.830594 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.827784 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5fb55fa3-ff25-4f04-8785-2c218ab4013d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.830594 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.827839 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5fb55fa3-ff25-4f04-8785-2c218ab4013d-federate-client-tls\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.830594 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.827904 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5fb55fa3-ff25-4f04-8785-2c218ab4013d-telemeter-client-tls\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.831596 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.831577 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fb55fa3-ff25-4f04-8785-2c218ab4013d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.836060 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.836038 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrrm\" (UniqueName: \"kubernetes.io/projected/5fb55fa3-ff25-4f04-8785-2c218ab4013d-kube-api-access-nqrrm\") pod \"telemeter-client-54876475cf-s9rxb\" (UID: \"5fb55fa3-ff25-4f04-8785-2c218ab4013d\") " pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:32.911382 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:32.911342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" Jul 06 20:38:33.051444 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:33.051359 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54876475cf-s9rxb"] Jul 06 20:38:33.055200 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:38:33.055162 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb55fa3_ff25_4f04_8785_2c218ab4013d.slice/crio-16417d694da22266b7d551d09e352bc62395669a2b359a270ca099d97a7e050e WatchSource:0}: Error finding container 16417d694da22266b7d551d09e352bc62395669a2b359a270ca099d97a7e050e: Status 404 returned error can't find the container with id 16417d694da22266b7d551d09e352bc62395669a2b359a270ca099d97a7e050e Jul 06 20:38:33.371293 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:33.371256 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" event={"ID":"5fb55fa3-ff25-4f04-8785-2c218ab4013d","Type":"ContainerStarted","Data":"16417d694da22266b7d551d09e352bc62395669a2b359a270ca099d97a7e050e"} Jul 06 20:38:35.381641 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:35.381603 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" event={"ID":"5fb55fa3-ff25-4f04-8785-2c218ab4013d","Type":"ContainerStarted","Data":"455d8a93932537d5a5d0fdfb5f1af7ce5ae41c124f2b9ff865e20e111439f071"} Jul 06 20:38:35.382066 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:35.381672 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" event={"ID":"5fb55fa3-ff25-4f04-8785-2c218ab4013d","Type":"ContainerStarted","Data":"cdc6764d6be6a2b6630e3f1dd682330e3c9005262933ffb5d8fe49e4ca0fad24"} Jul 06 20:38:35.382066 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:35.381704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" event={"ID":"5fb55fa3-ff25-4f04-8785-2c218ab4013d","Type":"ContainerStarted","Data":"795195918d7bcfa3eb2f4f798cbe577fa2eb82dfb15a9a15a9b5a6a2a40895e5"} Jul 06 20:38:35.414279 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:35.414231 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-54876475cf-s9rxb" podStartSLOduration=1.525327026 podStartE2EDuration="3.414216052s" totalImagesPullingTime="1.888889026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:38:35.413311019 +0000 UTC m=+206.543673611" watchObservedRunningTime="2026-07-06 20:38:35.414216052 +0000 UTC m=+206.544578643" Jul 06 20:38:55.542866 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:38:55.542827 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f9476f465-mwwnq"] Jul 06 20:39:20.562302 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.562243 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7f9476f465-mwwnq" podUID="9fa3c7a7-0c9a-4107-9244-829fe6c9918b" containerName="console" containerID="cri-o://88de0601cb9744686beec1bfd502bddb8c50d32480f770ed25a75cdb7e8bfec6" gracePeriod=15 Jul 06 20:39:20.798819 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.798793 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:39:20.879979 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.879942 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-oauth-config\") pod \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " Jul 06 20:39:20.880110 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.879990 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-service-ca\") pod \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " Jul 06 20:39:20.880110 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.880051 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-oauth-serving-cert\") pod \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " Jul 06 20:39:20.880110 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.880089 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-serving-cert\") pod \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " Jul 06 20:39:20.880263 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.880230 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfnwv\" (UniqueName: \"kubernetes.io/projected/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-kube-api-access-dfnwv\") pod \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " Jul 06 20:39:20.880319 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.880273 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-config\") pod \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " Jul 06 20:39:20.880368 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.880322 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-trusted-ca-bundle\") pod \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\" (UID: \"9fa3c7a7-0c9a-4107-9244-829fe6c9918b\") " Jul 06 20:39:20.880627 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.880481 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-service-ca" (OuterVolumeSpecName: "service-ca") pod "9fa3c7a7-0c9a-4107-9244-829fe6c9918b" (UID: "9fa3c7a7-0c9a-4107-9244-829fe6c9918b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:39:20.880627 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.880494 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9fa3c7a7-0c9a-4107-9244-829fe6c9918b" (UID: "9fa3c7a7-0c9a-4107-9244-829fe6c9918b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:39:20.880811 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.880654 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-oauth-serving-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:39:20.880811 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.880674 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-service-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:39:20.880811 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.880707 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-config" (OuterVolumeSpecName: "console-config") pod "9fa3c7a7-0c9a-4107-9244-829fe6c9918b" (UID: "9fa3c7a7-0c9a-4107-9244-829fe6c9918b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:39:20.880811 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.880764 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9fa3c7a7-0c9a-4107-9244-829fe6c9918b" (UID: "9fa3c7a7-0c9a-4107-9244-829fe6c9918b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:39:20.882290 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.882267 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9fa3c7a7-0c9a-4107-9244-829fe6c9918b" (UID: "9fa3c7a7-0c9a-4107-9244-829fe6c9918b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:39:20.882396 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.882377 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9fa3c7a7-0c9a-4107-9244-829fe6c9918b" (UID: "9fa3c7a7-0c9a-4107-9244-829fe6c9918b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:39:20.882511 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.882489 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-kube-api-access-dfnwv" (OuterVolumeSpecName: "kube-api-access-dfnwv") pod "9fa3c7a7-0c9a-4107-9244-829fe6c9918b" (UID: "9fa3c7a7-0c9a-4107-9244-829fe6c9918b"). InnerVolumeSpecName "kube-api-access-dfnwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:39:20.981131 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.981096 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-serving-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:39:20.981131 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.981123 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfnwv\" (UniqueName: \"kubernetes.io/projected/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-kube-api-access-dfnwv\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:39:20.981131 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.981134 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-config\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:39:20.981338 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.981142 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-trusted-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:39:20.981338 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:20.981150 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fa3c7a7-0c9a-4107-9244-829fe6c9918b-console-oauth-config\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:39:21.536075 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:21.536041 2579 generic.go:356] "Generic (PLEG): container finished" podID="9fa3c7a7-0c9a-4107-9244-829fe6c9918b" containerID="88de0601cb9744686beec1bfd502bddb8c50d32480f770ed25a75cdb7e8bfec6" exitCode=0 Jul 06 20:39:21.536226 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:21.536082 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9476f465-mwwnq" Jul 06 20:39:21.536226 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:21.536126 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9476f465-mwwnq" event={"ID":"9fa3c7a7-0c9a-4107-9244-829fe6c9918b","Type":"ContainerDied","Data":"88de0601cb9744686beec1bfd502bddb8c50d32480f770ed25a75cdb7e8bfec6"} Jul 06 20:39:21.536226 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:21.536167 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9476f465-mwwnq" event={"ID":"9fa3c7a7-0c9a-4107-9244-829fe6c9918b","Type":"ContainerDied","Data":"f2c4e65686648abafee0adfa9b3647f2c645e681514269e3409a781e4feb33c3"} Jul 06 20:39:21.536226 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:21.536186 2579 scope.go:117] "RemoveContainer" containerID="88de0601cb9744686beec1bfd502bddb8c50d32480f770ed25a75cdb7e8bfec6" Jul 06 20:39:21.545609 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:21.545592 2579 scope.go:117] "RemoveContainer" containerID="88de0601cb9744686beec1bfd502bddb8c50d32480f770ed25a75cdb7e8bfec6" Jul 06 20:39:21.545890 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:39:21.545872 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88de0601cb9744686beec1bfd502bddb8c50d32480f770ed25a75cdb7e8bfec6\": container with ID starting with 88de0601cb9744686beec1bfd502bddb8c50d32480f770ed25a75cdb7e8bfec6 not found: ID does not exist" containerID="88de0601cb9744686beec1bfd502bddb8c50d32480f770ed25a75cdb7e8bfec6" Jul 06 20:39:21.545935 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:21.545899 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88de0601cb9744686beec1bfd502bddb8c50d32480f770ed25a75cdb7e8bfec6"} err="failed to get container status \"88de0601cb9744686beec1bfd502bddb8c50d32480f770ed25a75cdb7e8bfec6\": rpc error: code = NotFound desc = could not find container \"88de0601cb9744686beec1bfd502bddb8c50d32480f770ed25a75cdb7e8bfec6\": container with ID starting with 88de0601cb9744686beec1bfd502bddb8c50d32480f770ed25a75cdb7e8bfec6 not found: ID does not exist" Jul 06 20:39:21.558589 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:21.558570 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f9476f465-mwwnq"] Jul 06 20:39:21.561941 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:21.561919 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f9476f465-mwwnq"] Jul 06 20:39:21.582859 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:39:21.582836 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fa3c7a7-0c9a-4107-9244-829fe6c9918b" path="/var/lib/kubelet/pods/9fa3c7a7-0c9a-4107-9244-829fe6c9918b/volumes" Jul 06 20:40:09.286604 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:40:09.286576 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 20:40:09.291942 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:40:09.291914 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 20:40:09.292166 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:40:09.292151 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 20:42:03.072705 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.072670 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-cjcjr"] Jul 06 20:42:03.073123 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.072912 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fa3c7a7-0c9a-4107-9244-829fe6c9918b" containerName="console" Jul 06 20:42:03.073123 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.072924 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fa3c7a7-0c9a-4107-9244-829fe6c9918b" containerName="console" Jul 06 20:42:03.073123 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.073004 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fa3c7a7-0c9a-4107-9244-829fe6c9918b" containerName="console" Jul 06 20:42:03.075632 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.075612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:42:03.077745 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.077725 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 06 20:42:03.077834 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.077748 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Jul 06 20:42:03.077834 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.077727 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-hxzwv\"" Jul 06 20:42:03.078234 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.078220 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 06 20:42:03.089093 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.089072 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-cjcjr"] Jul 06 20:42:03.090598 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.090577 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-xpfbd"] Jul 06 20:42:03.093421 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.093401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:42:03.095498 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.095479 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Jul 06 20:42:03.095616 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.095598 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-b844r\"" Jul 06 20:42:03.110624 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.110602 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-xpfbd"] Jul 06 20:42:03.122870 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.122849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert\") pod \"kserve-controller-manager-68bc99fcb8-cjcjr\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:42:03.122975 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.122883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l4hk\" (UniqueName: \"kubernetes.io/projected/955a9138-e4d9-48b6-83c0-946fea19b180-kube-api-access-2l4hk\") pod \"kserve-controller-manager-68bc99fcb8-cjcjr\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:42:03.223406 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.223383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlz4x\" (UniqueName: \"kubernetes.io/projected/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-kube-api-access-vlz4x\") pod \"llmisvc-controller-manager-67d95fdc48-xpfbd\" (UID: \"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:42:03.223524 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.223424 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert\") pod \"kserve-controller-manager-68bc99fcb8-cjcjr\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:42:03.223524 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.223442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xpfbd\" (UID: \"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:42:03.223524 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.223494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4hk\" (UniqueName: \"kubernetes.io/projected/955a9138-e4d9-48b6-83c0-946fea19b180-kube-api-access-2l4hk\") pod \"kserve-controller-manager-68bc99fcb8-cjcjr\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:42:03.223524 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:03.223512 2579 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 20:42:03.223677 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:03.223553 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert podName:955a9138-e4d9-48b6-83c0-946fea19b180 nodeName:}" failed. No retries permitted until 2026-07-06 20:42:03.723538933 +0000 UTC m=+414.853901504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert") pod "kserve-controller-manager-68bc99fcb8-cjcjr" (UID: "955a9138-e4d9-48b6-83c0-946fea19b180") : secret "kserve-webhook-server-cert" not found Jul 06 20:42:03.235769 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.235749 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l4hk\" (UniqueName: \"kubernetes.io/projected/955a9138-e4d9-48b6-83c0-946fea19b180-kube-api-access-2l4hk\") pod \"kserve-controller-manager-68bc99fcb8-cjcjr\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:42:03.324623 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.324566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xpfbd\" (UID: \"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:42:03.324623 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.324620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlz4x\" (UniqueName: \"kubernetes.io/projected/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-kube-api-access-vlz4x\") pod \"llmisvc-controller-manager-67d95fdc48-xpfbd\" (UID: \"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:42:03.324742 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:03.324699 2579 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 20:42:03.324779 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:03.324762 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert podName:5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e nodeName:}" failed. No retries permitted until 2026-07-06 20:42:03.824746504 +0000 UTC m=+414.955109091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert") pod "llmisvc-controller-manager-67d95fdc48-xpfbd" (UID: "5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e") : secret "llmisvc-webhook-server-cert" not found Jul 06 20:42:03.339080 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.339057 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlz4x\" (UniqueName: \"kubernetes.io/projected/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-kube-api-access-vlz4x\") pod \"llmisvc-controller-manager-67d95fdc48-xpfbd\" (UID: \"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:42:03.728716 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.728683 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert\") pod \"kserve-controller-manager-68bc99fcb8-cjcjr\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:42:03.728878 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:03.728799 2579 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 20:42:03.728878 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:03.728851 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert podName:955a9138-e4d9-48b6-83c0-946fea19b180 nodeName:}" failed. No retries permitted until 2026-07-06 20:42:04.728838944 +0000 UTC m=+415.859201515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert") pod "kserve-controller-manager-68bc99fcb8-cjcjr" (UID: "955a9138-e4d9-48b6-83c0-946fea19b180") : secret "kserve-webhook-server-cert" not found Jul 06 20:42:03.830169 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:03.830115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xpfbd\" (UID: \"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:42:03.830361 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:03.830281 2579 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 20:42:03.830361 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:03.830355 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert podName:5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e nodeName:}" failed. No retries permitted until 2026-07-06 20:42:04.830334448 +0000 UTC m=+415.960697032 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert") pod "llmisvc-controller-manager-67d95fdc48-xpfbd" (UID: "5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e") : secret "llmisvc-webhook-server-cert" not found Jul 06 20:42:04.736488 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:04.736446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert\") pod \"kserve-controller-manager-68bc99fcb8-cjcjr\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:42:04.736857 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:04.736591 2579 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 20:42:04.736857 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:04.736649 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert podName:955a9138-e4d9-48b6-83c0-946fea19b180 nodeName:}" failed. No retries permitted until 2026-07-06 20:42:06.736635761 +0000 UTC m=+417.866998332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert") pod "kserve-controller-manager-68bc99fcb8-cjcjr" (UID: "955a9138-e4d9-48b6-83c0-946fea19b180") : secret "kserve-webhook-server-cert" not found Jul 06 20:42:04.837570 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:04.837522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xpfbd\" (UID: \"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:42:04.837724 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:04.837662 2579 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 20:42:04.837768 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:04.837730 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert podName:5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e nodeName:}" failed. No retries permitted until 2026-07-06 20:42:06.837716216 +0000 UTC m=+417.968078792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert") pod "llmisvc-controller-manager-67d95fdc48-xpfbd" (UID: "5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e") : secret "llmisvc-webhook-server-cert" not found Jul 06 20:42:06.753426 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:06.753381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert\") pod \"kserve-controller-manager-68bc99fcb8-cjcjr\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:42:06.753866 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:06.753525 2579 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 20:42:06.753866 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:06.753579 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert podName:955a9138-e4d9-48b6-83c0-946fea19b180 nodeName:}" failed. No retries permitted until 2026-07-06 20:42:10.753565178 +0000 UTC m=+421.883927750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert") pod "kserve-controller-manager-68bc99fcb8-cjcjr" (UID: "955a9138-e4d9-48b6-83c0-946fea19b180") : secret "kserve-webhook-server-cert" not found Jul 06 20:42:06.853760 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:06.853731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xpfbd\" (UID: \"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:42:06.853919 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:06.853838 2579 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 20:42:06.853919 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:06.853884 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert podName:5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e nodeName:}" failed. No retries permitted until 2026-07-06 20:42:10.853872221 +0000 UTC m=+421.984234792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert") pod "llmisvc-controller-manager-67d95fdc48-xpfbd" (UID: "5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e") : secret "llmisvc-webhook-server-cert" not found Jul 06 20:42:10.782828 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:10.782791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert\") pod \"kserve-controller-manager-68bc99fcb8-cjcjr\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:42:10.783371 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:10.782948 2579 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 20:42:10.783371 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:10.783067 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert podName:955a9138-e4d9-48b6-83c0-946fea19b180 nodeName:}" failed. No retries permitted until 2026-07-06 20:42:18.783044597 +0000 UTC m=+429.913407168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert") pod "kserve-controller-manager-68bc99fcb8-cjcjr" (UID: "955a9138-e4d9-48b6-83c0-946fea19b180") : secret "kserve-webhook-server-cert" not found Jul 06 20:42:10.884198 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:10.884163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xpfbd\" (UID: \"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:42:10.884325 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:10.884281 2579 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 20:42:10.884366 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:10.884357 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert podName:5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e nodeName:}" failed. No retries permitted until 2026-07-06 20:42:18.884341405 +0000 UTC m=+430.014703980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert") pod "llmisvc-controller-manager-67d95fdc48-xpfbd" (UID: "5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e") : secret "llmisvc-webhook-server-cert" not found Jul 06 20:42:18.844157 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:18.844121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert\") pod \"kserve-controller-manager-68bc99fcb8-cjcjr\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:42:18.844636 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:18.844266 2579 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 20:42:18.844636 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:18.844321 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert podName:955a9138-e4d9-48b6-83c0-946fea19b180 nodeName:}" failed. No retries permitted until 2026-07-06 20:42:34.84430784 +0000 UTC m=+445.974670411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert") pod "kserve-controller-manager-68bc99fcb8-cjcjr" (UID: "955a9138-e4d9-48b6-83c0-946fea19b180") : secret "kserve-webhook-server-cert" not found Jul 06 20:42:18.945031 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:18.945001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xpfbd\" (UID: \"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:42:18.945171 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:18.945118 2579 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 20:42:18.945212 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:18.945172 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert podName:5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e nodeName:}" failed. No retries permitted until 2026-07-06 20:42:34.945159198 +0000 UTC m=+446.075521769 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert") pod "llmisvc-controller-manager-67d95fdc48-xpfbd" (UID: "5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e") : secret "llmisvc-webhook-server-cert" not found Jul 06 20:42:34.863551 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:34.863466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert\") pod \"kserve-controller-manager-68bc99fcb8-cjcjr\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:42:34.864005 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:34.863628 2579 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 20:42:34.864005 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:34.863705 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert podName:955a9138-e4d9-48b6-83c0-946fea19b180 nodeName:}" failed. No retries permitted until 2026-07-06 20:43:06.863687193 +0000 UTC m=+477.994049776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert") pod "kserve-controller-manager-68bc99fcb8-cjcjr" (UID: "955a9138-e4d9-48b6-83c0-946fea19b180") : secret "kserve-webhook-server-cert" not found Jul 06 20:42:34.964473 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:42:34.964435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xpfbd\" (UID: \"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:42:34.964605 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:34.964585 2579 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 20:42:34.964664 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:42:34.964658 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert podName:5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e nodeName:}" failed. No retries permitted until 2026-07-06 20:43:06.964643482 +0000 UTC m=+478.095006053 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert") pod "llmisvc-controller-manager-67d95fdc48-xpfbd" (UID: "5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e") : secret "llmisvc-webhook-server-cert" not found Jul 06 20:43:06.907849 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:06.907812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert\") pod \"kserve-controller-manager-68bc99fcb8-cjcjr\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:43:06.910224 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:06.910201 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert\") pod \"kserve-controller-manager-68bc99fcb8-cjcjr\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:43:06.987398 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:06.987373 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-hxzwv\"" Jul 06 20:43:06.996173 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:06.996158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:43:07.009195 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:07.009174 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xpfbd\" (UID: \"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:43:07.011315 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:07.011274 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xpfbd\" (UID: \"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:43:07.124464 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:07.124434 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-cjcjr"] Jul 06 20:43:07.127418 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:43:07.127377 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955a9138_e4d9_48b6_83c0_946fea19b180.slice/crio-997833729c1a88080086f444f44dd2653ad566c323e7a6d8b3b89bf8b98cf8df WatchSource:0}: Error finding container 997833729c1a88080086f444f44dd2653ad566c323e7a6d8b3b89bf8b98cf8df: Status 404 returned error can't find the container with id 997833729c1a88080086f444f44dd2653ad566c323e7a6d8b3b89bf8b98cf8df Jul 06 20:43:07.128766 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:07.128744 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:43:07.287324 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:07.287258 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" event={"ID":"955a9138-e4d9-48b6-83c0-946fea19b180","Type":"ContainerStarted","Data":"997833729c1a88080086f444f44dd2653ad566c323e7a6d8b3b89bf8b98cf8df"} Jul 06 20:43:07.304778 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:07.304759 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-b844r\"" Jul 06 20:43:07.312920 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:07.312902 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:43:07.437149 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:07.437116 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-xpfbd"] Jul 06 20:43:07.439642 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:43:07.439615 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5f9c265f_2aa2_4ea5_9a3e_c36b70ff234e.slice/crio-a2d173acd5dd2a6cbb2920430d4f3d4ca1ea67a78f9c224e8e083e30638d4de4 WatchSource:0}: Error finding container a2d173acd5dd2a6cbb2920430d4f3d4ca1ea67a78f9c224e8e083e30638d4de4: Status 404 returned error can't find the container with id a2d173acd5dd2a6cbb2920430d4f3d4ca1ea67a78f9c224e8e083e30638d4de4 Jul 06 20:43:08.293237 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:08.293162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" event={"ID":"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e","Type":"ContainerStarted","Data":"a2d173acd5dd2a6cbb2920430d4f3d4ca1ea67a78f9c224e8e083e30638d4de4"} Jul 06 20:43:11.308038 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:11.307993 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" event={"ID":"955a9138-e4d9-48b6-83c0-946fea19b180","Type":"ContainerStarted","Data":"cba9c38847f395f19bdc8009fa5a94972215e06f606d601685b6016c2f04c494"} Jul 06 20:43:11.308460 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:11.308084 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:43:11.309417 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:11.309393 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" event={"ID":"5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e","Type":"ContainerStarted","Data":"7b605a08eff9343b9226310f8acc46a7e801e4b87752301ceaaf3acf9808ae93"} Jul 06 20:43:11.309469 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:11.309449 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:43:11.330888 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:11.330845 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" podStartSLOduration=65.015943057 podStartE2EDuration="1m8.330831939s" totalImagesPullingTime="3.314888882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:43:11.329183479 +0000 UTC m=+482.459546072" watchObservedRunningTime="2026-07-06 20:43:11.330831939 +0000 UTC m=+482.461194532" Jul 06 20:43:11.346052 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:11.346009 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" podStartSLOduration=65.306971198 podStartE2EDuration="1m8.345993372s" totalImagesPullingTime="3.039022174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:43:11.345945965 +0000 UTC m=+482.476308557" watchObservedRunningTime="2026-07-06 20:43:11.345993372 +0000 UTC m=+482.476355959" Jul 06 20:43:42.315663 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:42.315619 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-xpfbd" Jul 06 20:43:42.318624 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:42.318603 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:43:43.776527 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:43.776491 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-cjcjr"] Jul 06 20:43:43.777004 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:43.776673 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" podUID="955a9138-e4d9-48b6-83c0-946fea19b180" containerName="manager" containerID="cri-o://cba9c38847f395f19bdc8009fa5a94972215e06f606d601685b6016c2f04c494" gracePeriod=10 Jul 06 20:43:43.805215 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:43.805188 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-vpqrl"] Jul 06 20:43:43.853484 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:43.853465 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-vpqrl"] Jul 06 20:43:43.853567 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:43.853552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-vpqrl" Jul 06 20:43:43.982448 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:43.982415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6643acdf-03d1-40d6-931e-956623a065ac-cert\") pod \"kserve-controller-manager-68bc99fcb8-vpqrl\" (UID: \"6643acdf-03d1-40d6-931e-956623a065ac\") " pod="kserve/kserve-controller-manager-68bc99fcb8-vpqrl" Jul 06 20:43:43.982593 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:43.982494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s49qm\" (UniqueName: \"kubernetes.io/projected/6643acdf-03d1-40d6-931e-956623a065ac-kube-api-access-s49qm\") pod \"kserve-controller-manager-68bc99fcb8-vpqrl\" (UID: \"6643acdf-03d1-40d6-931e-956623a065ac\") " pod="kserve/kserve-controller-manager-68bc99fcb8-vpqrl" Jul 06 20:43:44.047557 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.047504 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:43:44.083285 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.083260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s49qm\" (UniqueName: \"kubernetes.io/projected/6643acdf-03d1-40d6-931e-956623a065ac-kube-api-access-s49qm\") pod \"kserve-controller-manager-68bc99fcb8-vpqrl\" (UID: \"6643acdf-03d1-40d6-931e-956623a065ac\") " pod="kserve/kserve-controller-manager-68bc99fcb8-vpqrl" Jul 06 20:43:44.083413 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.083305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6643acdf-03d1-40d6-931e-956623a065ac-cert\") pod \"kserve-controller-manager-68bc99fcb8-vpqrl\" (UID: \"6643acdf-03d1-40d6-931e-956623a065ac\") " pod="kserve/kserve-controller-manager-68bc99fcb8-vpqrl" Jul 06 20:43:44.085540 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.085518 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6643acdf-03d1-40d6-931e-956623a065ac-cert\") pod \"kserve-controller-manager-68bc99fcb8-vpqrl\" (UID: \"6643acdf-03d1-40d6-931e-956623a065ac\") " pod="kserve/kserve-controller-manager-68bc99fcb8-vpqrl" Jul 06 20:43:44.103138 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.103105 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s49qm\" (UniqueName: \"kubernetes.io/projected/6643acdf-03d1-40d6-931e-956623a065ac-kube-api-access-s49qm\") pod \"kserve-controller-manager-68bc99fcb8-vpqrl\" (UID: \"6643acdf-03d1-40d6-931e-956623a065ac\") " pod="kserve/kserve-controller-manager-68bc99fcb8-vpqrl" Jul 06 20:43:44.184221 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.184190 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l4hk\" (UniqueName: \"kubernetes.io/projected/955a9138-e4d9-48b6-83c0-946fea19b180-kube-api-access-2l4hk\") pod \"955a9138-e4d9-48b6-83c0-946fea19b180\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " Jul 06 20:43:44.184221 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.184222 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert\") pod \"955a9138-e4d9-48b6-83c0-946fea19b180\" (UID: \"955a9138-e4d9-48b6-83c0-946fea19b180\") " Jul 06 20:43:44.186235 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.186206 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955a9138-e4d9-48b6-83c0-946fea19b180-kube-api-access-2l4hk" (OuterVolumeSpecName: "kube-api-access-2l4hk") pod "955a9138-e4d9-48b6-83c0-946fea19b180" (UID: "955a9138-e4d9-48b6-83c0-946fea19b180"). InnerVolumeSpecName "kube-api-access-2l4hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:43:44.186235 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.186227 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert" (OuterVolumeSpecName: "cert") pod "955a9138-e4d9-48b6-83c0-946fea19b180" (UID: "955a9138-e4d9-48b6-83c0-946fea19b180"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:43:44.216969 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.216932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-vpqrl" Jul 06 20:43:44.285612 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.285582 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l4hk\" (UniqueName: \"kubernetes.io/projected/955a9138-e4d9-48b6-83c0-946fea19b180-kube-api-access-2l4hk\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:43:44.285718 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.285618 2579 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/955a9138-e4d9-48b6-83c0-946fea19b180-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:43:44.362177 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.362102 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-vpqrl"] Jul 06 20:43:44.364582 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:43:44.364560 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6643acdf_03d1_40d6_931e_956623a065ac.slice/crio-8c76ae07ea8d05e2a8c562588ee0161f3bbd85d9054470f7a3e65b601a68d0bc WatchSource:0}: Error finding container 8c76ae07ea8d05e2a8c562588ee0161f3bbd85d9054470f7a3e65b601a68d0bc: Status 404 returned error can't find the container with id 8c76ae07ea8d05e2a8c562588ee0161f3bbd85d9054470f7a3e65b601a68d0bc Jul 06 20:43:44.422153 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.422113 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-vpqrl" event={"ID":"6643acdf-03d1-40d6-931e-956623a065ac","Type":"ContainerStarted","Data":"8c76ae07ea8d05e2a8c562588ee0161f3bbd85d9054470f7a3e65b601a68d0bc"} Jul 06 20:43:44.423203 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.423178 2579 generic.go:356] "Generic (PLEG): container finished" podID="955a9138-e4d9-48b6-83c0-946fea19b180" containerID="cba9c38847f395f19bdc8009fa5a94972215e06f606d601685b6016c2f04c494" exitCode=0 Jul 06 20:43:44.423269 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.423223 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" Jul 06 20:43:44.423314 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.423217 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" event={"ID":"955a9138-e4d9-48b6-83c0-946fea19b180","Type":"ContainerDied","Data":"cba9c38847f395f19bdc8009fa5a94972215e06f606d601685b6016c2f04c494"} Jul 06 20:43:44.423347 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.423335 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-cjcjr" event={"ID":"955a9138-e4d9-48b6-83c0-946fea19b180","Type":"ContainerDied","Data":"997833729c1a88080086f444f44dd2653ad566c323e7a6d8b3b89bf8b98cf8df"} Jul 06 20:43:44.423392 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.423358 2579 scope.go:117] "RemoveContainer" containerID="cba9c38847f395f19bdc8009fa5a94972215e06f606d601685b6016c2f04c494" Jul 06 20:43:44.431140 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.431125 2579 scope.go:117] "RemoveContainer" containerID="cba9c38847f395f19bdc8009fa5a94972215e06f606d601685b6016c2f04c494" Jul 06 20:43:44.431378 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:43:44.431358 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba9c38847f395f19bdc8009fa5a94972215e06f606d601685b6016c2f04c494\": container with ID starting with cba9c38847f395f19bdc8009fa5a94972215e06f606d601685b6016c2f04c494 not found: ID does not exist" containerID="cba9c38847f395f19bdc8009fa5a94972215e06f606d601685b6016c2f04c494" Jul 06 20:43:44.431441 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.431388 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba9c38847f395f19bdc8009fa5a94972215e06f606d601685b6016c2f04c494"} err="failed to get container status \"cba9c38847f395f19bdc8009fa5a94972215e06f606d601685b6016c2f04c494\": rpc error: code = NotFound desc = could not find container \"cba9c38847f395f19bdc8009fa5a94972215e06f606d601685b6016c2f04c494\": container with ID starting with cba9c38847f395f19bdc8009fa5a94972215e06f606d601685b6016c2f04c494 not found: ID does not exist" Jul 06 20:43:44.444464 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.444444 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-cjcjr"] Jul 06 20:43:44.451489 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:44.451471 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-68bc99fcb8-cjcjr"] Jul 06 20:43:45.429307 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:45.429265 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc99fcb8-vpqrl" event={"ID":"6643acdf-03d1-40d6-931e-956623a065ac","Type":"ContainerStarted","Data":"7b435ff7bcc1c8b4bb6a2abc243066414d2f6e14a4d16f6f4316c0a6b6e5edc6"} Jul 06 20:43:45.429714 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:45.429406 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-68bc99fcb8-vpqrl" Jul 06 20:43:45.446991 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:45.446928 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-68bc99fcb8-vpqrl" podStartSLOduration=2.037123191 podStartE2EDuration="2.446915369s" totalImagesPullingTime="409.792178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:43:45.446100369 +0000 UTC m=+516.576462962" watchObservedRunningTime="2026-07-06 20:43:45.446915369 +0000 UTC m=+516.577277962" Jul 06 20:43:45.583139 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:43:45.583113 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="955a9138-e4d9-48b6-83c0-946fea19b180" path="/var/lib/kubelet/pods/955a9138-e4d9-48b6-83c0-946fea19b180/volumes" Jul 06 20:44:16.436828 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:16.436797 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-68bc99fcb8-vpqrl" Jul 06 20:44:17.625691 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:17.625657 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-m4885"] Jul 06 20:44:17.626061 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:17.625915 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="955a9138-e4d9-48b6-83c0-946fea19b180" containerName="manager" Jul 06 20:44:17.626061 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:17.625925 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="955a9138-e4d9-48b6-83c0-946fea19b180" containerName="manager" Jul 06 20:44:17.626061 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:17.625983 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="955a9138-e4d9-48b6-83c0-946fea19b180" containerName="manager" Jul 06 20:44:17.629901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:17.629884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-m4885" Jul 06 20:44:17.642905 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:17.642883 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Jul 06 20:44:17.648546 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:17.648533 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-k7kxf\"" Jul 06 20:44:17.663362 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:17.663337 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-m4885"] Jul 06 20:44:17.722751 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:17.722724 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmccp\" (UniqueName: \"kubernetes.io/projected/9bca35b4-13eb-4eef-bbf2-6d424a750967-kube-api-access-xmccp\") pod \"odh-model-controller-84b54b7f97-m4885\" (UID: \"9bca35b4-13eb-4eef-bbf2-6d424a750967\") " pod="kserve/odh-model-controller-84b54b7f97-m4885" Jul 06 20:44:17.722851 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:17.722756 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bca35b4-13eb-4eef-bbf2-6d424a750967-cert\") pod \"odh-model-controller-84b54b7f97-m4885\" (UID: \"9bca35b4-13eb-4eef-bbf2-6d424a750967\") " pod="kserve/odh-model-controller-84b54b7f97-m4885" Jul 06 20:44:17.824113 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:17.824084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bca35b4-13eb-4eef-bbf2-6d424a750967-cert\") pod \"odh-model-controller-84b54b7f97-m4885\" (UID: \"9bca35b4-13eb-4eef-bbf2-6d424a750967\") " pod="kserve/odh-model-controller-84b54b7f97-m4885" Jul 06 20:44:17.824257 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:44:17.824237 2579 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 06 20:44:17.824340 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:17.824240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmccp\" (UniqueName: \"kubernetes.io/projected/9bca35b4-13eb-4eef-bbf2-6d424a750967-kube-api-access-xmccp\") pod \"odh-model-controller-84b54b7f97-m4885\" (UID: \"9bca35b4-13eb-4eef-bbf2-6d424a750967\") " pod="kserve/odh-model-controller-84b54b7f97-m4885" Jul 06 20:44:17.824340 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:44:17.824301 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bca35b4-13eb-4eef-bbf2-6d424a750967-cert podName:9bca35b4-13eb-4eef-bbf2-6d424a750967 nodeName:}" failed. No retries permitted until 2026-07-06 20:44:18.324286647 +0000 UTC m=+549.454649223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9bca35b4-13eb-4eef-bbf2-6d424a750967-cert") pod "odh-model-controller-84b54b7f97-m4885" (UID: "9bca35b4-13eb-4eef-bbf2-6d424a750967") : secret "odh-model-controller-webhook-cert" not found Jul 06 20:44:17.840718 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:17.840693 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmccp\" (UniqueName: \"kubernetes.io/projected/9bca35b4-13eb-4eef-bbf2-6d424a750967-kube-api-access-xmccp\") pod \"odh-model-controller-84b54b7f97-m4885\" (UID: \"9bca35b4-13eb-4eef-bbf2-6d424a750967\") " pod="kserve/odh-model-controller-84b54b7f97-m4885" Jul 06 20:44:18.329137 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:18.329104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bca35b4-13eb-4eef-bbf2-6d424a750967-cert\") pod \"odh-model-controller-84b54b7f97-m4885\" (UID: \"9bca35b4-13eb-4eef-bbf2-6d424a750967\") " pod="kserve/odh-model-controller-84b54b7f97-m4885" Jul 06 20:44:18.331374 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:18.331345 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bca35b4-13eb-4eef-bbf2-6d424a750967-cert\") pod \"odh-model-controller-84b54b7f97-m4885\" (UID: \"9bca35b4-13eb-4eef-bbf2-6d424a750967\") " pod="kserve/odh-model-controller-84b54b7f97-m4885" Jul 06 20:44:18.539808 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:18.539780 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-m4885" Jul 06 20:44:18.662390 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:18.662361 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-m4885"] Jul 06 20:44:18.666017 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:44:18.665991 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bca35b4_13eb_4eef_bbf2_6d424a750967.slice/crio-60041c2bd16caa95a7d6b145ca3bf6548cc0e139ae42f6d9b19623130e389ed6 WatchSource:0}: Error finding container 60041c2bd16caa95a7d6b145ca3bf6548cc0e139ae42f6d9b19623130e389ed6: Status 404 returned error can't find the container with id 60041c2bd16caa95a7d6b145ca3bf6548cc0e139ae42f6d9b19623130e389ed6 Jul 06 20:44:19.541080 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:19.541040 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-m4885" event={"ID":"9bca35b4-13eb-4eef-bbf2-6d424a750967","Type":"ContainerStarted","Data":"60041c2bd16caa95a7d6b145ca3bf6548cc0e139ae42f6d9b19623130e389ed6"} Jul 06 20:44:22.552882 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:22.552847 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-m4885" event={"ID":"9bca35b4-13eb-4eef-bbf2-6d424a750967","Type":"ContainerStarted","Data":"60bafbb752192f0aaf4ec23c263266cc5434c178c57c12f9d66a7eb1e5a4d290"} Jul 06 20:44:22.553367 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:22.552980 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-m4885" Jul 06 20:44:22.572620 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:22.572575 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-m4885" podStartSLOduration=2.686045996 podStartE2EDuration="5.572561116s" totalImagesPullingTime="2.88651512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:44:22.571162218 +0000 UTC m=+553.701524813" watchObservedRunningTime="2026-07-06 20:44:22.572561116 +0000 UTC m=+553.702923755" Jul 06 20:44:33.558190 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:44:33.558161 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-m4885" Jul 06 20:45:09.309845 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:45:09.309807 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 20:45:09.312398 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:45:09.312375 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 20:46:05.299431 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.299394 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz"] Jul 06 20:46:05.302756 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.302739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.305042 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.305018 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\"" Jul 06 20:46:05.305237 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.305208 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 06 20:46:05.305379 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.305047 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\"" Jul 06 20:46:05.305379 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.305077 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 06 20:46:05.305379 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.305080 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 06 20:46:05.305589 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.305572 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 06 20:46:05.305660 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.305628 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-b6hg7\"" Jul 06 20:46:05.312975 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.312939 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz"] Jul 06 20:46:05.445261 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.445227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzskp\" (UniqueName: \"kubernetes.io/projected/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kube-api-access-tzskp\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.445261 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.445261 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e0c7517-edbb-448f-9869-9e7df46d3e8f-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.445455 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.445291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.445455 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.445369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.445455 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.445420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e0c7517-edbb-448f-9869-9e7df46d3e8f-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.546207 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.546168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e0c7517-edbb-448f-9869-9e7df46d3e8f-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.546381 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.546220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzskp\" (UniqueName: \"kubernetes.io/projected/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kube-api-access-tzskp\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.546381 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.546242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e0c7517-edbb-448f-9869-9e7df46d3e8f-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.546381 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.546271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.546381 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.546297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.546381 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:46:05.546335 2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found Jul 06 20:46:05.546570 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:46:05.546418 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0c7517-edbb-448f-9869-9e7df46d3e8f-proxy-tls podName:0e0c7517-edbb-448f-9869-9e7df46d3e8f nodeName:}" failed. No retries permitted until 2026-07-06 20:46:06.046397161 +0000 UTC m=+657.176759735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0e0c7517-edbb-448f-9869-9e7df46d3e8f-proxy-tls") pod "isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" (UID: "0e0c7517-edbb-448f-9869-9e7df46d3e8f") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found Jul 06 20:46:05.546724 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.546706 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.546988 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.546941 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e0c7517-edbb-448f-9869-9e7df46d3e8f-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.547105 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.547023 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:05.557631 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:05.557573 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzskp\" (UniqueName: \"kubernetes.io/projected/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kube-api-access-tzskp\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:06.051087 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:06.051058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e0c7517-edbb-448f-9869-9e7df46d3e8f-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:06.053308 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:06.053281 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e0c7517-edbb-448f-9869-9e7df46d3e8f-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:06.213967 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:06.213931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:06.342907 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:06.342835 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz"] Jul 06 20:46:06.345608 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:46:06.345582 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0c7517_edbb_448f_9869_9e7df46d3e8f.slice/crio-b8e45473b129d5fbca8fc5d6d4d03dd074b972fe61c59253754f3ada16d5bc31 WatchSource:0}: Error finding container b8e45473b129d5fbca8fc5d6d4d03dd074b972fe61c59253754f3ada16d5bc31: Status 404 returned error can't find the container with id b8e45473b129d5fbca8fc5d6d4d03dd074b972fe61c59253754f3ada16d5bc31 Jul 06 20:46:06.898169 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:06.898121 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" event={"ID":"0e0c7517-edbb-448f-9869-9e7df46d3e8f","Type":"ContainerStarted","Data":"b8e45473b129d5fbca8fc5d6d4d03dd074b972fe61c59253754f3ada16d5bc31"} Jul 06 20:46:09.421847 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:09.421821 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 06 20:46:09.911733 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:09.911696 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" event={"ID":"0e0c7517-edbb-448f-9869-9e7df46d3e8f","Type":"ContainerStarted","Data":"cad883826a1759d48c198e4aac04314986d2e9315b3c988537fda57c48fc0f88"} Jul 06 20:46:13.928344 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:13.928309 2579 generic.go:356] "Generic (PLEG): container finished" podID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerID="cad883826a1759d48c198e4aac04314986d2e9315b3c988537fda57c48fc0f88" exitCode=0 Jul 06 20:46:13.928705 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:13.928367 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" event={"ID":"0e0c7517-edbb-448f-9869-9e7df46d3e8f","Type":"ContainerDied","Data":"cad883826a1759d48c198e4aac04314986d2e9315b3c988537fda57c48fc0f88"} Jul 06 20:46:26.988393 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:26.988311 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" event={"ID":"0e0c7517-edbb-448f-9869-9e7df46d3e8f","Type":"ContainerStarted","Data":"a348631a69ad6ac1d1e31673b3ee49c33da69db5867254b6a53822ef08e5df99"} Jul 06 20:46:30.003783 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:30.003741 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" event={"ID":"0e0c7517-edbb-448f-9869-9e7df46d3e8f","Type":"ContainerStarted","Data":"f3c1d7cee5b7e4190ceff64f7688022adb95566cd80941db0598550792192b34"} Jul 06 20:46:30.004254 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:30.003950 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:30.004254 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:30.004009 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:30.005305 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:30.005276 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 06 20:46:30.028164 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:30.028114 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" podStartSLOduration=3.279304091 podStartE2EDuration="25.028100696s" totalImagesPullingTime="18.047110122s" totalInitContainerRuntime="3.701686483s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:46:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:46:30.026322825 +0000 UTC m=+681.156685418" watchObservedRunningTime="2026-07-06 20:46:30.028100696 +0000 UTC m=+681.158463288" Jul 06 20:46:31.007144 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:31.007107 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 06 20:46:36.011053 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:36.011024 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:46:36.011452 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:36.011427 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 06 20:46:46.011532 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:46.011489 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 06 20:46:56.011784 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:46:56.011743 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 06 20:47:06.011513 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:06.011418 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 06 20:47:16.012073 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:16.012030 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 06 20:47:24.622107 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:24.622076 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w"] Jul 06 20:47:24.628088 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:24.628058 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:47:24.630515 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:24.630492 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-85420-serving-cert\"" Jul 06 20:47:24.630620 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:24.630512 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-85420-kube-rbac-proxy-sar-config\"" Jul 06 20:47:24.634915 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:24.634752 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w"] Jul 06 20:47:24.728754 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:24.728728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0602ee6b-3f5e-4601-b2c0-e93758035dfa-proxy-tls\") pod \"switch-graph-85420-5d67fb78c6-cxj5w\" (UID: \"0602ee6b-3f5e-4601-b2c0-e93758035dfa\") " pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:47:24.728901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:24.728791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0602ee6b-3f5e-4601-b2c0-e93758035dfa-openshift-service-ca-bundle\") pod \"switch-graph-85420-5d67fb78c6-cxj5w\" (UID: \"0602ee6b-3f5e-4601-b2c0-e93758035dfa\") " pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:47:24.829434 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:24.829405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0602ee6b-3f5e-4601-b2c0-e93758035dfa-openshift-service-ca-bundle\") pod \"switch-graph-85420-5d67fb78c6-cxj5w\" (UID: \"0602ee6b-3f5e-4601-b2c0-e93758035dfa\") " pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:47:24.829554 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:24.829436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0602ee6b-3f5e-4601-b2c0-e93758035dfa-proxy-tls\") pod \"switch-graph-85420-5d67fb78c6-cxj5w\" (UID: \"0602ee6b-3f5e-4601-b2c0-e93758035dfa\") " pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:47:24.829554 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:47:24.829541 2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-85420-serving-cert: secret "switch-graph-85420-serving-cert" not found Jul 06 20:47:24.829621 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:47:24.829609 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0602ee6b-3f5e-4601-b2c0-e93758035dfa-proxy-tls podName:0602ee6b-3f5e-4601-b2c0-e93758035dfa nodeName:}" failed. No retries permitted until 2026-07-06 20:47:25.329593307 +0000 UTC m=+736.459955878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0602ee6b-3f5e-4601-b2c0-e93758035dfa-proxy-tls") pod "switch-graph-85420-5d67fb78c6-cxj5w" (UID: "0602ee6b-3f5e-4601-b2c0-e93758035dfa") : secret "switch-graph-85420-serving-cert" not found Jul 06 20:47:24.830042 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:24.830023 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0602ee6b-3f5e-4601-b2c0-e93758035dfa-openshift-service-ca-bundle\") pod \"switch-graph-85420-5d67fb78c6-cxj5w\" (UID: \"0602ee6b-3f5e-4601-b2c0-e93758035dfa\") " pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:47:25.335054 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:25.335026 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0602ee6b-3f5e-4601-b2c0-e93758035dfa-proxy-tls\") pod \"switch-graph-85420-5d67fb78c6-cxj5w\" (UID: \"0602ee6b-3f5e-4601-b2c0-e93758035dfa\") " pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:47:25.337384 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:25.337361 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0602ee6b-3f5e-4601-b2c0-e93758035dfa-proxy-tls\") pod \"switch-graph-85420-5d67fb78c6-cxj5w\" (UID: \"0602ee6b-3f5e-4601-b2c0-e93758035dfa\") " pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:47:25.538857 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:25.538834 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:47:25.664997 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:25.664973 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w"] Jul 06 20:47:25.667145 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:47:25.667120 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0602ee6b_3f5e_4601_b2c0_e93758035dfa.slice/crio-b7ff690845fdbfff0c019a873a584230e2bbefba7b9b7c67298844d5bba853ca WatchSource:0}: Error finding container b7ff690845fdbfff0c019a873a584230e2bbefba7b9b7c67298844d5bba853ca: Status 404 returned error can't find the container with id b7ff690845fdbfff0c019a873a584230e2bbefba7b9b7c67298844d5bba853ca Jul 06 20:47:26.011830 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:26.011796 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 06 20:47:26.200626 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:26.200596 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" event={"ID":"0602ee6b-3f5e-4601-b2c0-e93758035dfa","Type":"ContainerStarted","Data":"b7ff690845fdbfff0c019a873a584230e2bbefba7b9b7c67298844d5bba853ca"} Jul 06 20:47:28.210319 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:28.210285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" event={"ID":"0602ee6b-3f5e-4601-b2c0-e93758035dfa","Type":"ContainerStarted","Data":"f6a9631948d779ffa7d383e684e1ed5be561c9be310b5865f1894b419ab41612"} Jul 06 20:47:28.210679 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:28.210419 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:47:28.231897 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:28.231849 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" podStartSLOduration=2.112319379 podStartE2EDuration="4.231835523s" totalImagesPullingTime="2.119516144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:47:28.23110945 +0000 UTC m=+739.361472045" watchObservedRunningTime="2026-07-06 20:47:28.231835523 +0000 UTC m=+739.362198115" Jul 06 20:47:34.220561 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:34.220531 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:47:34.846406 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:34.846373 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w"] Jul 06 20:47:34.846598 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:34.846573 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" podUID="0602ee6b-3f5e-4601-b2c0-e93758035dfa" containerName="switch-graph-85420" containerID="cri-o://f6a9631948d779ffa7d383e684e1ed5be561c9be310b5865f1894b419ab41612" gracePeriod=30 Jul 06 20:47:36.012688 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:36.012649 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:47:39.218529 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:39.218474 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" podUID="0602ee6b-3f5e-4601-b2c0-e93758035dfa" containerName="switch-graph-85420" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:47:44.218418 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:44.218352 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" podUID="0602ee6b-3f5e-4601-b2c0-e93758035dfa" containerName="switch-graph-85420" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:47:49.218847 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:49.218802 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" podUID="0602ee6b-3f5e-4601-b2c0-e93758035dfa" containerName="switch-graph-85420" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:47:49.219242 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:49.218919 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:47:54.218609 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:54.218562 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" podUID="0602ee6b-3f5e-4601-b2c0-e93758035dfa" containerName="switch-graph-85420" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:47:59.217829 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:47:59.217783 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" podUID="0602ee6b-3f5e-4601-b2c0-e93758035dfa" containerName="switch-graph-85420" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:48:04.218866 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.218804 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" podUID="0602ee6b-3f5e-4601-b2c0-e93758035dfa" containerName="switch-graph-85420" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:48:04.596759 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.596682 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht"] Jul 06 20:48:04.602592 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.602569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" Jul 06 20:48:04.604763 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.604742 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Jul 06 20:48:04.605132 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.605105 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Jul 06 20:48:04.608846 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.608822 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht"] Jul 06 20:48:04.698518 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.698476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440f131a-4c58-44d7-8fd7-f3279ce6d988-openshift-service-ca-bundle\") pod \"model-chainer-7dc7f99b6d-qtdht\" (UID: \"440f131a-4c58-44d7-8fd7-f3279ce6d988\") " pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" Jul 06 20:48:04.698699 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.698539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440f131a-4c58-44d7-8fd7-f3279ce6d988-proxy-tls\") pod \"model-chainer-7dc7f99b6d-qtdht\" (UID: \"440f131a-4c58-44d7-8fd7-f3279ce6d988\") " pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" Jul 06 20:48:04.799455 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.799420 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440f131a-4c58-44d7-8fd7-f3279ce6d988-openshift-service-ca-bundle\") pod \"model-chainer-7dc7f99b6d-qtdht\" (UID: \"440f131a-4c58-44d7-8fd7-f3279ce6d988\") " pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" Jul 06 20:48:04.799455 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.799457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440f131a-4c58-44d7-8fd7-f3279ce6d988-proxy-tls\") pod \"model-chainer-7dc7f99b6d-qtdht\" (UID: \"440f131a-4c58-44d7-8fd7-f3279ce6d988\") " pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" Jul 06 20:48:04.800179 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.800158 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440f131a-4c58-44d7-8fd7-f3279ce6d988-openshift-service-ca-bundle\") pod \"model-chainer-7dc7f99b6d-qtdht\" (UID: \"440f131a-4c58-44d7-8fd7-f3279ce6d988\") " pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" Jul 06 20:48:04.801795 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.801774 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440f131a-4c58-44d7-8fd7-f3279ce6d988-proxy-tls\") pod \"model-chainer-7dc7f99b6d-qtdht\" (UID: \"440f131a-4c58-44d7-8fd7-f3279ce6d988\") " pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" Jul 06 20:48:04.913232 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.913196 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" Jul 06 20:48:04.989323 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:04.989303 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:48:05.000850 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.000828 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0602ee6b-3f5e-4601-b2c0-e93758035dfa-proxy-tls\") pod \"0602ee6b-3f5e-4601-b2c0-e93758035dfa\" (UID: \"0602ee6b-3f5e-4601-b2c0-e93758035dfa\") " Jul 06 20:48:05.000970 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.000886 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0602ee6b-3f5e-4601-b2c0-e93758035dfa-openshift-service-ca-bundle\") pod \"0602ee6b-3f5e-4601-b2c0-e93758035dfa\" (UID: \"0602ee6b-3f5e-4601-b2c0-e93758035dfa\") " Jul 06 20:48:05.001270 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.001249 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0602ee6b-3f5e-4601-b2c0-e93758035dfa-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0602ee6b-3f5e-4601-b2c0-e93758035dfa" (UID: "0602ee6b-3f5e-4601-b2c0-e93758035dfa"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:48:05.003563 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.003543 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0602ee6b-3f5e-4601-b2c0-e93758035dfa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0602ee6b-3f5e-4601-b2c0-e93758035dfa" (UID: "0602ee6b-3f5e-4601-b2c0-e93758035dfa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:48:05.039382 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.039290 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht"] Jul 06 20:48:05.041767 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:48:05.041740 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440f131a_4c58_44d7_8fd7_f3279ce6d988.slice/crio-8934db374a74910f70aa6c4b646f228e6bd7918919b42e089d8c434cf8e2f051 WatchSource:0}: Error finding container 8934db374a74910f70aa6c4b646f228e6bd7918919b42e089d8c434cf8e2f051: Status 404 returned error can't find the container with id 8934db374a74910f70aa6c4b646f228e6bd7918919b42e089d8c434cf8e2f051 Jul 06 20:48:05.101435 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.101395 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0602ee6b-3f5e-4601-b2c0-e93758035dfa-proxy-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:48:05.101435 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.101416 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0602ee6b-3f5e-4601-b2c0-e93758035dfa-openshift-service-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:48:05.361340 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.361255 2579 generic.go:356] "Generic (PLEG): container finished" podID="0602ee6b-3f5e-4601-b2c0-e93758035dfa" containerID="f6a9631948d779ffa7d383e684e1ed5be561c9be310b5865f1894b419ab41612" exitCode=0 Jul 06 20:48:05.361340 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.361302 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" Jul 06 20:48:05.361340 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.361308 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" event={"ID":"0602ee6b-3f5e-4601-b2c0-e93758035dfa","Type":"ContainerDied","Data":"f6a9631948d779ffa7d383e684e1ed5be561c9be310b5865f1894b419ab41612"} Jul 06 20:48:05.361828 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.361393 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w" event={"ID":"0602ee6b-3f5e-4601-b2c0-e93758035dfa","Type":"ContainerDied","Data":"b7ff690845fdbfff0c019a873a584230e2bbefba7b9b7c67298844d5bba853ca"} Jul 06 20:48:05.361828 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.361433 2579 scope.go:117] "RemoveContainer" containerID="f6a9631948d779ffa7d383e684e1ed5be561c9be310b5865f1894b419ab41612" Jul 06 20:48:05.363261 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.363222 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" event={"ID":"440f131a-4c58-44d7-8fd7-f3279ce6d988","Type":"ContainerStarted","Data":"47ade216023c7e9a6735d3361426ccb4cf1d2f9842cc5a7f458829402074f01d"} Jul 06 20:48:05.363405 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.363283 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" event={"ID":"440f131a-4c58-44d7-8fd7-f3279ce6d988","Type":"ContainerStarted","Data":"8934db374a74910f70aa6c4b646f228e6bd7918919b42e089d8c434cf8e2f051"} Jul 06 20:48:05.363405 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.363319 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" Jul 06 20:48:05.376184 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.370698 2579 scope.go:117] "RemoveContainer" containerID="f6a9631948d779ffa7d383e684e1ed5be561c9be310b5865f1894b419ab41612" Jul 06 20:48:05.376639 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:48:05.376607 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a9631948d779ffa7d383e684e1ed5be561c9be310b5865f1894b419ab41612\": container with ID starting with f6a9631948d779ffa7d383e684e1ed5be561c9be310b5865f1894b419ab41612 not found: ID does not exist" containerID="f6a9631948d779ffa7d383e684e1ed5be561c9be310b5865f1894b419ab41612" Jul 06 20:48:05.376752 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.376645 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a9631948d779ffa7d383e684e1ed5be561c9be310b5865f1894b419ab41612"} err="failed to get container status \"f6a9631948d779ffa7d383e684e1ed5be561c9be310b5865f1894b419ab41612\": rpc error: code = NotFound desc = could not find container \"f6a9631948d779ffa7d383e684e1ed5be561c9be310b5865f1894b419ab41612\": container with ID starting with f6a9631948d779ffa7d383e684e1ed5be561c9be310b5865f1894b419ab41612 not found: ID does not exist" Jul 06 20:48:05.383397 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.383346 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" podStartSLOduration=1.3833344269999999 podStartE2EDuration="1.383334427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:05.382415174 +0000 UTC m=+776.512777768" watchObservedRunningTime="2026-07-06 20:48:05.383334427 +0000 UTC m=+776.513697020" Jul 06 20:48:05.395923 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.395888 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w"] Jul 06 20:48:05.404877 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.404851 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-85420-5d67fb78c6-cxj5w"] Jul 06 20:48:05.583659 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:05.583637 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0602ee6b-3f5e-4601-b2c0-e93758035dfa" path="/var/lib/kubelet/pods/0602ee6b-3f5e-4601-b2c0-e93758035dfa/volumes" Jul 06 20:48:11.374048 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:11.374013 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" Jul 06 20:48:14.684474 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:14.684442 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht"] Jul 06 20:48:14.684829 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:14.684621 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" podUID="440f131a-4c58-44d7-8fd7-f3279ce6d988" containerName="model-chainer" containerID="cri-o://47ade216023c7e9a6735d3361426ccb4cf1d2f9842cc5a7f458829402074f01d" gracePeriod=30 Jul 06 20:48:14.828141 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:14.828111 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz"] Jul 06 20:48:14.828429 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:14.828402 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kserve-container" containerID="cri-o://a348631a69ad6ac1d1e31673b3ee49c33da69db5867254b6a53822ef08e5df99" gracePeriod=30 Jul 06 20:48:14.828531 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:14.828430 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kube-rbac-proxy" containerID="cri-o://f3c1d7cee5b7e4190ceff64f7688022adb95566cd80941db0598550792192b34" gracePeriod=30 Jul 06 20:48:15.406818 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:15.406785 2579 generic.go:356] "Generic (PLEG): container finished" podID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerID="f3c1d7cee5b7e4190ceff64f7688022adb95566cd80941db0598550792192b34" exitCode=2 Jul 06 20:48:15.406986 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:15.406826 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" event={"ID":"0e0c7517-edbb-448f-9869-9e7df46d3e8f","Type":"ContainerDied","Data":"f3c1d7cee5b7e4190ceff64f7688022adb95566cd80941db0598550792192b34"} Jul 06 20:48:16.007775 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:16.007736 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused" Jul 06 20:48:16.012068 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:16.012046 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 06 20:48:16.372376 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:16.372337 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" podUID="440f131a-4c58-44d7-8fd7-f3279ce6d988" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:48:18.575248 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.575226 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:48:18.603442 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.603417 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kube-rbac-proxy-sar-config\") pod \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " Jul 06 20:48:18.603571 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.603479 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kserve-provision-location\") pod \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " Jul 06 20:48:18.603571 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.603511 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e0c7517-edbb-448f-9869-9e7df46d3e8f-cabundle-cert\") pod \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " Jul 06 20:48:18.603571 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.603550 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzskp\" (UniqueName: \"kubernetes.io/projected/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kube-api-access-tzskp\") pod \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " Jul 06 20:48:18.603731 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.603605 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e0c7517-edbb-448f-9869-9e7df46d3e8f-proxy-tls\") pod \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\" (UID: \"0e0c7517-edbb-448f-9869-9e7df46d3e8f\") " Jul 06 20:48:18.603874 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.603847 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e0c7517-edbb-448f-9869-9e7df46d3e8f" (UID: "0e0c7517-edbb-448f-9869-9e7df46d3e8f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:48:18.603946 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.603850 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e0c7517-edbb-448f-9869-9e7df46d3e8f" (UID: "0e0c7517-edbb-448f-9869-9e7df46d3e8f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:48:18.603946 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.603886 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0c7517-edbb-448f-9869-9e7df46d3e8f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e0c7517-edbb-448f-9869-9e7df46d3e8f" (UID: "0e0c7517-edbb-448f-9869-9e7df46d3e8f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:48:18.605602 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.605572 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e0c7517-edbb-448f-9869-9e7df46d3e8f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e0c7517-edbb-448f-9869-9e7df46d3e8f" (UID: "0e0c7517-edbb-448f-9869-9e7df46d3e8f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:48:18.605683 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.605619 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kube-api-access-tzskp" (OuterVolumeSpecName: "kube-api-access-tzskp") pod "0e0c7517-edbb-448f-9869-9e7df46d3e8f" (UID: "0e0c7517-edbb-448f-9869-9e7df46d3e8f"). InnerVolumeSpecName "kube-api-access-tzskp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:48:18.704884 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.704833 2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:48:18.704884 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.704853 2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kserve-provision-location\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:48:18.704884 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.704862 2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e0c7517-edbb-448f-9869-9e7df46d3e8f-cabundle-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:48:18.704884 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.704870 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzskp\" (UniqueName: \"kubernetes.io/projected/0e0c7517-edbb-448f-9869-9e7df46d3e8f-kube-api-access-tzskp\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:48:18.704884 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:18.704878 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e0c7517-edbb-448f-9869-9e7df46d3e8f-proxy-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:48:19.423648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.423612 2579 generic.go:356] "Generic (PLEG): container finished" podID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerID="a348631a69ad6ac1d1e31673b3ee49c33da69db5867254b6a53822ef08e5df99" exitCode=0 Jul 06 20:48:19.423814 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.423667 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" event={"ID":"0e0c7517-edbb-448f-9869-9e7df46d3e8f","Type":"ContainerDied","Data":"a348631a69ad6ac1d1e31673b3ee49c33da69db5867254b6a53822ef08e5df99"} Jul 06 20:48:19.423814 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.423690 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" Jul 06 20:48:19.423814 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.423723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz" event={"ID":"0e0c7517-edbb-448f-9869-9e7df46d3e8f","Type":"ContainerDied","Data":"b8e45473b129d5fbca8fc5d6d4d03dd074b972fe61c59253754f3ada16d5bc31"} Jul 06 20:48:19.423814 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.423754 2579 scope.go:117] "RemoveContainer" containerID="f3c1d7cee5b7e4190ceff64f7688022adb95566cd80941db0598550792192b34" Jul 06 20:48:19.433373 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.433353 2579 scope.go:117] "RemoveContainer" containerID="a348631a69ad6ac1d1e31673b3ee49c33da69db5867254b6a53822ef08e5df99" Jul 06 20:48:19.441479 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.441459 2579 scope.go:117] "RemoveContainer" containerID="cad883826a1759d48c198e4aac04314986d2e9315b3c988537fda57c48fc0f88" Jul 06 20:48:19.449017 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.448790 2579 scope.go:117] "RemoveContainer" containerID="f3c1d7cee5b7e4190ceff64f7688022adb95566cd80941db0598550792192b34" Jul 06 20:48:19.449141 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:48:19.449120 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c1d7cee5b7e4190ceff64f7688022adb95566cd80941db0598550792192b34\": container with ID starting with f3c1d7cee5b7e4190ceff64f7688022adb95566cd80941db0598550792192b34 not found: ID does not exist" containerID="f3c1d7cee5b7e4190ceff64f7688022adb95566cd80941db0598550792192b34" Jul 06 20:48:19.449193 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.449150 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c1d7cee5b7e4190ceff64f7688022adb95566cd80941db0598550792192b34"} err="failed to get container status \"f3c1d7cee5b7e4190ceff64f7688022adb95566cd80941db0598550792192b34\": rpc error: code = NotFound desc = could not find container \"f3c1d7cee5b7e4190ceff64f7688022adb95566cd80941db0598550792192b34\": container with ID starting with f3c1d7cee5b7e4190ceff64f7688022adb95566cd80941db0598550792192b34 not found: ID does not exist" Jul 06 20:48:19.449193 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.449168 2579 scope.go:117] "RemoveContainer" containerID="a348631a69ad6ac1d1e31673b3ee49c33da69db5867254b6a53822ef08e5df99" Jul 06 20:48:19.449443 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:48:19.449424 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a348631a69ad6ac1d1e31673b3ee49c33da69db5867254b6a53822ef08e5df99\": container with ID starting with a348631a69ad6ac1d1e31673b3ee49c33da69db5867254b6a53822ef08e5df99 not found: ID does not exist" containerID="a348631a69ad6ac1d1e31673b3ee49c33da69db5867254b6a53822ef08e5df99" Jul 06 20:48:19.449486 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.449449 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a348631a69ad6ac1d1e31673b3ee49c33da69db5867254b6a53822ef08e5df99"} err="failed to get container status \"a348631a69ad6ac1d1e31673b3ee49c33da69db5867254b6a53822ef08e5df99\": rpc error: code = NotFound desc = could not find container \"a348631a69ad6ac1d1e31673b3ee49c33da69db5867254b6a53822ef08e5df99\": container with ID starting with a348631a69ad6ac1d1e31673b3ee49c33da69db5867254b6a53822ef08e5df99 not found: ID does not exist" Jul 06 20:48:19.449486 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.449467 2579 scope.go:117] "RemoveContainer" containerID="cad883826a1759d48c198e4aac04314986d2e9315b3c988537fda57c48fc0f88" Jul 06 20:48:19.449752 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:48:19.449731 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad883826a1759d48c198e4aac04314986d2e9315b3c988537fda57c48fc0f88\": container with ID starting with cad883826a1759d48c198e4aac04314986d2e9315b3c988537fda57c48fc0f88 not found: ID does not exist" containerID="cad883826a1759d48c198e4aac04314986d2e9315b3c988537fda57c48fc0f88" Jul 06 20:48:19.449811 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.449756 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad883826a1759d48c198e4aac04314986d2e9315b3c988537fda57c48fc0f88"} err="failed to get container status \"cad883826a1759d48c198e4aac04314986d2e9315b3c988537fda57c48fc0f88\": rpc error: code = NotFound desc = could not find container \"cad883826a1759d48c198e4aac04314986d2e9315b3c988537fda57c48fc0f88\": container with ID starting with cad883826a1759d48c198e4aac04314986d2e9315b3c988537fda57c48fc0f88 not found: ID does not exist" Jul 06 20:48:19.449811 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.449761 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz"] Jul 06 20:48:19.456254 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.456232 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6664c8d89d-hd2nz"] Jul 06 20:48:19.583901 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:19.583876 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" path="/var/lib/kubelet/pods/0e0c7517-edbb-448f-9869-9e7df46d3e8f/volumes" Jul 06 20:48:21.372456 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:21.372409 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" podUID="440f131a-4c58-44d7-8fd7-f3279ce6d988" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:48:26.371604 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:26.371572 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" podUID="440f131a-4c58-44d7-8fd7-f3279ce6d988" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:48:26.372036 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:26.371661 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" Jul 06 20:48:31.372329 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:31.372290 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" podUID="440f131a-4c58-44d7-8fd7-f3279ce6d988" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:48:35.103383 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.103295 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r"] Jul 06 20:48:35.103732 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.103676 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0602ee6b-3f5e-4601-b2c0-e93758035dfa" containerName="switch-graph-85420" Jul 06 20:48:35.103732 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.103691 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0602ee6b-3f5e-4601-b2c0-e93758035dfa" containerName="switch-graph-85420" Jul 06 20:48:35.103732 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.103710 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kube-rbac-proxy" Jul 06 20:48:35.103732 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.103717 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kube-rbac-proxy" Jul 06 20:48:35.103877 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.103738 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="storage-initializer" Jul 06 20:48:35.103877 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.103747 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="storage-initializer" Jul 06 20:48:35.103877 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.103755 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kserve-container" Jul 06 20:48:35.103877 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.103762 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kserve-container" Jul 06 20:48:35.103877 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.103819 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0602ee6b-3f5e-4601-b2c0-e93758035dfa" containerName="switch-graph-85420" Jul 06 20:48:35.103877 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.103832 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kserve-container" Jul 06 20:48:35.103877 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.103840 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e0c7517-edbb-448f-9869-9e7df46d3e8f" containerName="kube-rbac-proxy" Jul 06 20:48:35.108448 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.108431 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" Jul 06 20:48:35.110862 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.110839 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-87d4d-serving-cert\"" Jul 06 20:48:35.111410 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.111395 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-87d4d-kube-rbac-proxy-sar-config\"" Jul 06 20:48:35.119655 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.119634 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r"] Jul 06 20:48:35.220060 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.220022 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/498c6f24-ef95-4fe0-9ae9-b8d93076b086-proxy-tls\") pod \"switch-graph-87d4d-6dfd87976f-kww7r\" (UID: \"498c6f24-ef95-4fe0-9ae9-b8d93076b086\") " pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" Jul 06 20:48:35.220221 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.220087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498c6f24-ef95-4fe0-9ae9-b8d93076b086-openshift-service-ca-bundle\") pod \"switch-graph-87d4d-6dfd87976f-kww7r\" (UID: \"498c6f24-ef95-4fe0-9ae9-b8d93076b086\") " pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" Jul 06 20:48:35.321409 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.321379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/498c6f24-ef95-4fe0-9ae9-b8d93076b086-proxy-tls\") pod \"switch-graph-87d4d-6dfd87976f-kww7r\" (UID: \"498c6f24-ef95-4fe0-9ae9-b8d93076b086\") " pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" Jul 06 20:48:35.321409 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.321409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498c6f24-ef95-4fe0-9ae9-b8d93076b086-openshift-service-ca-bundle\") pod \"switch-graph-87d4d-6dfd87976f-kww7r\" (UID: \"498c6f24-ef95-4fe0-9ae9-b8d93076b086\") " pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" Jul 06 20:48:35.322075 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.322056 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498c6f24-ef95-4fe0-9ae9-b8d93076b086-openshift-service-ca-bundle\") pod \"switch-graph-87d4d-6dfd87976f-kww7r\" (UID: \"498c6f24-ef95-4fe0-9ae9-b8d93076b086\") " pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" Jul 06 20:48:35.323820 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.323794 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/498c6f24-ef95-4fe0-9ae9-b8d93076b086-proxy-tls\") pod \"switch-graph-87d4d-6dfd87976f-kww7r\" (UID: \"498c6f24-ef95-4fe0-9ae9-b8d93076b086\") " pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" Jul 06 20:48:35.417925 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.417869 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" Jul 06 20:48:35.541003 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:35.540974 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r"] Jul 06 20:48:35.543839 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:48:35.543816 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498c6f24_ef95_4fe0_9ae9_b8d93076b086.slice/crio-afa5293302e73e2f1d097e78520a22a0516c598e93285e3d9e0ff381de739d07 WatchSource:0}: Error finding container afa5293302e73e2f1d097e78520a22a0516c598e93285e3d9e0ff381de739d07: Status 404 returned error can't find the container with id afa5293302e73e2f1d097e78520a22a0516c598e93285e3d9e0ff381de739d07 Jul 06 20:48:36.372451 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:36.372411 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" podUID="440f131a-4c58-44d7-8fd7-f3279ce6d988" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:48:36.481400 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:36.481365 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" event={"ID":"498c6f24-ef95-4fe0-9ae9-b8d93076b086","Type":"ContainerStarted","Data":"32879fc9086bc656a4ceebc3d4f0039a51a1dcfd968b0bb76610756d60c92e53"} Jul 06 20:48:36.481546 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:36.481429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" event={"ID":"498c6f24-ef95-4fe0-9ae9-b8d93076b086","Type":"ContainerStarted","Data":"afa5293302e73e2f1d097e78520a22a0516c598e93285e3d9e0ff381de739d07"} Jul 06 20:48:36.481546 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:36.481475 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" Jul 06 20:48:36.500848 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:36.500798 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" podStartSLOduration=1.500780376 podStartE2EDuration="1.500780376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:36.500273003 +0000 UTC m=+807.630635595" watchObservedRunningTime="2026-07-06 20:48:36.500780376 +0000 UTC m=+807.631143046" Jul 06 20:48:41.372499 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:41.372450 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" podUID="440f131a-4c58-44d7-8fd7-f3279ce6d988" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:48:42.490367 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:42.490340 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" Jul 06 20:48:44.824131 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:44.824110 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" Jul 06 20:48:44.886983 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:44.886943 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440f131a-4c58-44d7-8fd7-f3279ce6d988-openshift-service-ca-bundle\") pod \"440f131a-4c58-44d7-8fd7-f3279ce6d988\" (UID: \"440f131a-4c58-44d7-8fd7-f3279ce6d988\") " Jul 06 20:48:44.887091 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:44.886996 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440f131a-4c58-44d7-8fd7-f3279ce6d988-proxy-tls\") pod \"440f131a-4c58-44d7-8fd7-f3279ce6d988\" (UID: \"440f131a-4c58-44d7-8fd7-f3279ce6d988\") " Jul 06 20:48:44.887270 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:44.887246 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440f131a-4c58-44d7-8fd7-f3279ce6d988-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "440f131a-4c58-44d7-8fd7-f3279ce6d988" (UID: "440f131a-4c58-44d7-8fd7-f3279ce6d988"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:48:44.888804 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:44.888781 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440f131a-4c58-44d7-8fd7-f3279ce6d988-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "440f131a-4c58-44d7-8fd7-f3279ce6d988" (UID: "440f131a-4c58-44d7-8fd7-f3279ce6d988"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:48:44.987659 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:44.987607 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440f131a-4c58-44d7-8fd7-f3279ce6d988-openshift-service-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:48:44.987659 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:44.987628 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440f131a-4c58-44d7-8fd7-f3279ce6d988-proxy-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:48:45.516398 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:45.516365 2579 generic.go:356] "Generic (PLEG): container finished" podID="440f131a-4c58-44d7-8fd7-f3279ce6d988" containerID="47ade216023c7e9a6735d3361426ccb4cf1d2f9842cc5a7f458829402074f01d" exitCode=0 Jul 06 20:48:45.516549 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:45.516412 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" Jul 06 20:48:45.516549 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:45.516440 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" event={"ID":"440f131a-4c58-44d7-8fd7-f3279ce6d988","Type":"ContainerDied","Data":"47ade216023c7e9a6735d3361426ccb4cf1d2f9842cc5a7f458829402074f01d"} Jul 06 20:48:45.516549 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:45.516501 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht" event={"ID":"440f131a-4c58-44d7-8fd7-f3279ce6d988","Type":"ContainerDied","Data":"8934db374a74910f70aa6c4b646f228e6bd7918919b42e089d8c434cf8e2f051"} Jul 06 20:48:45.516549 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:45.516533 2579 scope.go:117] "RemoveContainer" containerID="47ade216023c7e9a6735d3361426ccb4cf1d2f9842cc5a7f458829402074f01d" Jul 06 20:48:45.526031 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:45.526011 2579 scope.go:117] "RemoveContainer" containerID="47ade216023c7e9a6735d3361426ccb4cf1d2f9842cc5a7f458829402074f01d" Jul 06 20:48:45.526322 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:48:45.526304 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ade216023c7e9a6735d3361426ccb4cf1d2f9842cc5a7f458829402074f01d\": container with ID starting with 47ade216023c7e9a6735d3361426ccb4cf1d2f9842cc5a7f458829402074f01d not found: ID does not exist" containerID="47ade216023c7e9a6735d3361426ccb4cf1d2f9842cc5a7f458829402074f01d" Jul 06 20:48:45.526376 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:45.526328 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ade216023c7e9a6735d3361426ccb4cf1d2f9842cc5a7f458829402074f01d"} err="failed to get container status \"47ade216023c7e9a6735d3361426ccb4cf1d2f9842cc5a7f458829402074f01d\": rpc error: code = NotFound desc = could not find container \"47ade216023c7e9a6735d3361426ccb4cf1d2f9842cc5a7f458829402074f01d\": container with ID starting with 47ade216023c7e9a6735d3361426ccb4cf1d2f9842cc5a7f458829402074f01d not found: ID does not exist" Jul 06 20:48:45.543737 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:45.543709 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht"] Jul 06 20:48:45.562510 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:45.562482 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7dc7f99b6d-qtdht"] Jul 06 20:48:45.586188 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:48:45.584703 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="440f131a-4c58-44d7-8fd7-f3279ce6d988" path="/var/lib/kubelet/pods/440f131a-4c58-44d7-8fd7-f3279ce6d988/volumes" Jul 06 20:49:14.880443 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:14.880413 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn"] Jul 06 20:49:14.880857 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:14.880698 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440f131a-4c58-44d7-8fd7-f3279ce6d988" containerName="model-chainer" Jul 06 20:49:14.880857 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:14.880709 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="440f131a-4c58-44d7-8fd7-f3279ce6d988" containerName="model-chainer" Jul 06 20:49:14.880857 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:14.880762 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="440f131a-4c58-44d7-8fd7-f3279ce6d988" containerName="model-chainer" Jul 06 20:49:14.884941 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:14.884924 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" Jul 06 20:49:14.887281 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:14.887262 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-dfad7-kube-rbac-proxy-sar-config\"" Jul 06 20:49:14.887380 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:14.887276 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-dfad7-serving-cert\"" Jul 06 20:49:14.898181 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:14.898158 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn"] Jul 06 20:49:14.994228 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:14.994191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a72cc3a0-b146-4295-add1-c67bdead913a-proxy-tls\") pod \"sequence-graph-dfad7-5f5d97c8c5-9l9jn\" (UID: \"a72cc3a0-b146-4295-add1-c67bdead913a\") " pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" Jul 06 20:49:14.994368 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:14.994258 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a72cc3a0-b146-4295-add1-c67bdead913a-openshift-service-ca-bundle\") pod \"sequence-graph-dfad7-5f5d97c8c5-9l9jn\" (UID: \"a72cc3a0-b146-4295-add1-c67bdead913a\") " pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" Jul 06 20:49:15.094623 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:15.094594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a72cc3a0-b146-4295-add1-c67bdead913a-proxy-tls\") pod \"sequence-graph-dfad7-5f5d97c8c5-9l9jn\" (UID: \"a72cc3a0-b146-4295-add1-c67bdead913a\") " pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" Jul 06 20:49:15.094776 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:15.094639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a72cc3a0-b146-4295-add1-c67bdead913a-openshift-service-ca-bundle\") pod \"sequence-graph-dfad7-5f5d97c8c5-9l9jn\" (UID: \"a72cc3a0-b146-4295-add1-c67bdead913a\") " pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" Jul 06 20:49:15.095217 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:15.095195 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a72cc3a0-b146-4295-add1-c67bdead913a-openshift-service-ca-bundle\") pod \"sequence-graph-dfad7-5f5d97c8c5-9l9jn\" (UID: \"a72cc3a0-b146-4295-add1-c67bdead913a\") " pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" Jul 06 20:49:15.096775 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:15.096754 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a72cc3a0-b146-4295-add1-c67bdead913a-proxy-tls\") pod \"sequence-graph-dfad7-5f5d97c8c5-9l9jn\" (UID: \"a72cc3a0-b146-4295-add1-c67bdead913a\") " pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" Jul 06 20:49:15.194390 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:15.194325 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" Jul 06 20:49:15.317460 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:15.317433 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn"] Jul 06 20:49:15.320075 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:49:15.320050 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72cc3a0_b146_4295_add1_c67bdead913a.slice/crio-150059a05f0025a812c16b1c6ae23aa64782febb54a622757b03641e9f7f1b31 WatchSource:0}: Error finding container 150059a05f0025a812c16b1c6ae23aa64782febb54a622757b03641e9f7f1b31: Status 404 returned error can't find the container with id 150059a05f0025a812c16b1c6ae23aa64782febb54a622757b03641e9f7f1b31 Jul 06 20:49:15.620813 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:15.620781 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" event={"ID":"a72cc3a0-b146-4295-add1-c67bdead913a","Type":"ContainerStarted","Data":"523ff0abe7249639d8eda0db3a10b77e92a0bf5c277b3fe0fe7fd75f1f38455e"} Jul 06 20:49:15.620987 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:15.620826 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" event={"ID":"a72cc3a0-b146-4295-add1-c67bdead913a","Type":"ContainerStarted","Data":"150059a05f0025a812c16b1c6ae23aa64782febb54a622757b03641e9f7f1b31"} Jul 06 20:49:15.620987 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:15.620908 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" Jul 06 20:49:15.637863 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:15.637819 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" podStartSLOduration=1.637804263 podStartE2EDuration="1.637804263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:15.637598874 +0000 UTC m=+846.767961466" watchObservedRunningTime="2026-07-06 20:49:15.637804263 +0000 UTC m=+846.768166854" Jul 06 20:49:21.629533 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:49:21.629504 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" Jul 06 20:50:09.335293 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:50:09.335184 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 20:50:09.336542 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:50:09.336519 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 20:55:09.359873 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:55:09.359840 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 20:55:09.362441 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:55:09.360437 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 20:56:49.752132 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:56:49.752091 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r"] Jul 06 20:56:49.752687 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:56:49.752274 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" podUID="498c6f24-ef95-4fe0-9ae9-b8d93076b086" containerName="switch-graph-87d4d" containerID="cri-o://32879fc9086bc656a4ceebc3d4f0039a51a1dcfd968b0bb76610756d60c92e53" gracePeriod=30 Jul 06 20:56:52.489751 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:56:52.489699 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" podUID="498c6f24-ef95-4fe0-9ae9-b8d93076b086" containerName="switch-graph-87d4d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:56:57.489508 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:56:57.489449 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" podUID="498c6f24-ef95-4fe0-9ae9-b8d93076b086" containerName="switch-graph-87d4d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:57:02.489773 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:02.489726 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" podUID="498c6f24-ef95-4fe0-9ae9-b8d93076b086" containerName="switch-graph-87d4d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:57:02.490300 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:02.489894 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" Jul 06 20:57:07.488870 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:07.488823 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" podUID="498c6f24-ef95-4fe0-9ae9-b8d93076b086" containerName="switch-graph-87d4d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:57:12.488868 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:12.488825 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" podUID="498c6f24-ef95-4fe0-9ae9-b8d93076b086" containerName="switch-graph-87d4d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:57:17.490027 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:17.489982 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" podUID="498c6f24-ef95-4fe0-9ae9-b8d93076b086" containerName="switch-graph-87d4d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:57:19.926795 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:19.926769 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" Jul 06 20:57:19.977108 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:19.977087 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498c6f24-ef95-4fe0-9ae9-b8d93076b086-openshift-service-ca-bundle\") pod \"498c6f24-ef95-4fe0-9ae9-b8d93076b086\" (UID: \"498c6f24-ef95-4fe0-9ae9-b8d93076b086\") " Jul 06 20:57:19.977202 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:19.977147 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/498c6f24-ef95-4fe0-9ae9-b8d93076b086-proxy-tls\") pod \"498c6f24-ef95-4fe0-9ae9-b8d93076b086\" (UID: \"498c6f24-ef95-4fe0-9ae9-b8d93076b086\") " Jul 06 20:57:19.977466 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:19.977443 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498c6f24-ef95-4fe0-9ae9-b8d93076b086-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "498c6f24-ef95-4fe0-9ae9-b8d93076b086" (UID: "498c6f24-ef95-4fe0-9ae9-b8d93076b086"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:57:19.978948 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:19.978930 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/498c6f24-ef95-4fe0-9ae9-b8d93076b086-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "498c6f24-ef95-4fe0-9ae9-b8d93076b086" (UID: "498c6f24-ef95-4fe0-9ae9-b8d93076b086"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:57:20.077843 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:20.077792 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/498c6f24-ef95-4fe0-9ae9-b8d93076b086-proxy-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:57:20.077843 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:20.077813 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498c6f24-ef95-4fe0-9ae9-b8d93076b086-openshift-service-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:57:20.369996 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:20.369943 2579 generic.go:356] "Generic (PLEG): container finished" podID="498c6f24-ef95-4fe0-9ae9-b8d93076b086" containerID="32879fc9086bc656a4ceebc3d4f0039a51a1dcfd968b0bb76610756d60c92e53" exitCode=0 Jul 06 20:57:20.370118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:20.370017 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" Jul 06 20:57:20.370118 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:20.370059 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" event={"ID":"498c6f24-ef95-4fe0-9ae9-b8d93076b086","Type":"ContainerDied","Data":"32879fc9086bc656a4ceebc3d4f0039a51a1dcfd968b0bb76610756d60c92e53"} Jul 06 20:57:20.370222 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:20.370122 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r" event={"ID":"498c6f24-ef95-4fe0-9ae9-b8d93076b086","Type":"ContainerDied","Data":"afa5293302e73e2f1d097e78520a22a0516c598e93285e3d9e0ff381de739d07"} Jul 06 20:57:20.370222 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:20.370178 2579 scope.go:117] "RemoveContainer" containerID="32879fc9086bc656a4ceebc3d4f0039a51a1dcfd968b0bb76610756d60c92e53" Jul 06 20:57:20.379543 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:20.379522 2579 scope.go:117] "RemoveContainer" containerID="32879fc9086bc656a4ceebc3d4f0039a51a1dcfd968b0bb76610756d60c92e53" Jul 06 20:57:20.379841 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:57:20.379815 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32879fc9086bc656a4ceebc3d4f0039a51a1dcfd968b0bb76610756d60c92e53\": container with ID starting with 32879fc9086bc656a4ceebc3d4f0039a51a1dcfd968b0bb76610756d60c92e53 not found: ID does not exist" containerID="32879fc9086bc656a4ceebc3d4f0039a51a1dcfd968b0bb76610756d60c92e53" Jul 06 20:57:20.379982 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:20.379846 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32879fc9086bc656a4ceebc3d4f0039a51a1dcfd968b0bb76610756d60c92e53"} err="failed to get container status \"32879fc9086bc656a4ceebc3d4f0039a51a1dcfd968b0bb76610756d60c92e53\": rpc error: code = NotFound desc = could not find container \"32879fc9086bc656a4ceebc3d4f0039a51a1dcfd968b0bb76610756d60c92e53\": container with ID starting with 32879fc9086bc656a4ceebc3d4f0039a51a1dcfd968b0bb76610756d60c92e53 not found: ID does not exist" Jul 06 20:57:20.393505 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:20.393477 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r"] Jul 06 20:57:20.400365 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:20.400344 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-87d4d-6dfd87976f-kww7r"] Jul 06 20:57:21.583172 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:21.583146 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="498c6f24-ef95-4fe0-9ae9-b8d93076b086" path="/var/lib/kubelet/pods/498c6f24-ef95-4fe0-9ae9-b8d93076b086/volumes" Jul 06 20:57:29.536560 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:29.536528 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn"] Jul 06 20:57:29.537103 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:29.536755 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" podUID="a72cc3a0-b146-4295-add1-c67bdead913a" containerName="sequence-graph-dfad7" containerID="cri-o://523ff0abe7249639d8eda0db3a10b77e92a0bf5c277b3fe0fe7fd75f1f38455e" gracePeriod=30 Jul 06 20:57:31.627871 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:31.627829 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" podUID="a72cc3a0-b146-4295-add1-c67bdead913a" containerName="sequence-graph-dfad7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:57:36.628613 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:36.628568 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" podUID="a72cc3a0-b146-4295-add1-c67bdead913a" containerName="sequence-graph-dfad7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:57:41.627582 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:41.627542 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" podUID="a72cc3a0-b146-4295-add1-c67bdead913a" containerName="sequence-graph-dfad7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:57:41.627981 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:41.627652 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" Jul 06 20:57:46.628256 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:46.628219 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" podUID="a72cc3a0-b146-4295-add1-c67bdead913a" containerName="sequence-graph-dfad7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:57:51.627826 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:51.627784 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" podUID="a72cc3a0-b146-4295-add1-c67bdead913a" containerName="sequence-graph-dfad7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:57:56.627744 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:57:56.627701 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" podUID="a72cc3a0-b146-4295-add1-c67bdead913a" containerName="sequence-graph-dfad7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:58:00.051490 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.051466 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn"] Jul 06 20:58:00.051772 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.051739 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="498c6f24-ef95-4fe0-9ae9-b8d93076b086" containerName="switch-graph-87d4d" Jul 06 20:58:00.051772 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.051755 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="498c6f24-ef95-4fe0-9ae9-b8d93076b086" containerName="switch-graph-87d4d" Jul 06 20:58:00.051913 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.051827 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="498c6f24-ef95-4fe0-9ae9-b8d93076b086" containerName="switch-graph-87d4d" Jul 06 20:58:00.054514 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.054487 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:00.056469 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.056447 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-305f4-kube-rbac-proxy-sar-config\"" Jul 06 20:58:00.056648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.056634 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-305f4-serving-cert\"" Jul 06 20:58:00.063606 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.063583 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn"] Jul 06 20:58:00.150694 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.150653 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-openshift-service-ca-bundle\") pod \"ensemble-graph-305f4-d67d47567-krvxn\" (UID: \"b2eccb3f-95f3-4ffb-aeb4-4604950b7089\") " pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:00.150942 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.150920 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-proxy-tls\") pod \"ensemble-graph-305f4-d67d47567-krvxn\" (UID: \"b2eccb3f-95f3-4ffb-aeb4-4604950b7089\") " pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:00.175574 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.175553 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" Jul 06 20:58:00.251643 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.251619 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a72cc3a0-b146-4295-add1-c67bdead913a-proxy-tls\") pod \"a72cc3a0-b146-4295-add1-c67bdead913a\" (UID: \"a72cc3a0-b146-4295-add1-c67bdead913a\") " Jul 06 20:58:00.251756 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.251647 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a72cc3a0-b146-4295-add1-c67bdead913a-openshift-service-ca-bundle\") pod \"a72cc3a0-b146-4295-add1-c67bdead913a\" (UID: \"a72cc3a0-b146-4295-add1-c67bdead913a\") " Jul 06 20:58:00.251756 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.251720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-proxy-tls\") pod \"ensemble-graph-305f4-d67d47567-krvxn\" (UID: \"b2eccb3f-95f3-4ffb-aeb4-4604950b7089\") " pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:00.251863 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.251764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-openshift-service-ca-bundle\") pod \"ensemble-graph-305f4-d67d47567-krvxn\" (UID: \"b2eccb3f-95f3-4ffb-aeb4-4604950b7089\") " pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:00.251915 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:58:00.251867 2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-305f4-serving-cert: secret "ensemble-graph-305f4-serving-cert" not found Jul 06 20:58:00.252003 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:58:00.251931 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-proxy-tls podName:b2eccb3f-95f3-4ffb-aeb4-4604950b7089 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:00.751912507 +0000 UTC m=+1371.882275094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-proxy-tls") pod "ensemble-graph-305f4-d67d47567-krvxn" (UID: "b2eccb3f-95f3-4ffb-aeb4-4604950b7089") : secret "ensemble-graph-305f4-serving-cert" not found Jul 06 20:58:00.252113 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.252054 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72cc3a0-b146-4295-add1-c67bdead913a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a72cc3a0-b146-4295-add1-c67bdead913a" (UID: "a72cc3a0-b146-4295-add1-c67bdead913a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:58:00.252323 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.252300 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-openshift-service-ca-bundle\") pod \"ensemble-graph-305f4-d67d47567-krvxn\" (UID: \"b2eccb3f-95f3-4ffb-aeb4-4604950b7089\") " pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:00.253470 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.253453 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72cc3a0-b146-4295-add1-c67bdead913a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a72cc3a0-b146-4295-add1-c67bdead913a" (UID: "a72cc3a0-b146-4295-add1-c67bdead913a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:58:00.352892 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.352872 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a72cc3a0-b146-4295-add1-c67bdead913a-proxy-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:58:00.352892 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.352892 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a72cc3a0-b146-4295-add1-c67bdead913a-openshift-service-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:58:00.518659 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.518621 2579 generic.go:356] "Generic (PLEG): container finished" podID="a72cc3a0-b146-4295-add1-c67bdead913a" containerID="523ff0abe7249639d8eda0db3a10b77e92a0bf5c277b3fe0fe7fd75f1f38455e" exitCode=0 Jul 06 20:58:00.519046 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.519021 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" event={"ID":"a72cc3a0-b146-4295-add1-c67bdead913a","Type":"ContainerDied","Data":"523ff0abe7249639d8eda0db3a10b77e92a0bf5c277b3fe0fe7fd75f1f38455e"} Jul 06 20:58:00.519248 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.519231 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" event={"ID":"a72cc3a0-b146-4295-add1-c67bdead913a","Type":"ContainerDied","Data":"150059a05f0025a812c16b1c6ae23aa64782febb54a622757b03641e9f7f1b31"} Jul 06 20:58:00.519355 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.519244 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn" Jul 06 20:58:00.519466 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.519274 2579 scope.go:117] "RemoveContainer" containerID="523ff0abe7249639d8eda0db3a10b77e92a0bf5c277b3fe0fe7fd75f1f38455e" Jul 06 20:58:00.528748 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.528726 2579 scope.go:117] "RemoveContainer" containerID="523ff0abe7249639d8eda0db3a10b77e92a0bf5c277b3fe0fe7fd75f1f38455e" Jul 06 20:58:00.529047 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:58:00.529021 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523ff0abe7249639d8eda0db3a10b77e92a0bf5c277b3fe0fe7fd75f1f38455e\": container with ID starting with 523ff0abe7249639d8eda0db3a10b77e92a0bf5c277b3fe0fe7fd75f1f38455e not found: ID does not exist" containerID="523ff0abe7249639d8eda0db3a10b77e92a0bf5c277b3fe0fe7fd75f1f38455e" Jul 06 20:58:00.529150 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.529051 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523ff0abe7249639d8eda0db3a10b77e92a0bf5c277b3fe0fe7fd75f1f38455e"} err="failed to get container status \"523ff0abe7249639d8eda0db3a10b77e92a0bf5c277b3fe0fe7fd75f1f38455e\": rpc error: code = NotFound desc = could not find container \"523ff0abe7249639d8eda0db3a10b77e92a0bf5c277b3fe0fe7fd75f1f38455e\": container with ID starting with 523ff0abe7249639d8eda0db3a10b77e92a0bf5c277b3fe0fe7fd75f1f38455e not found: ID does not exist" Jul 06 20:58:00.542331 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.542303 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn"] Jul 06 20:58:00.549335 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.549311 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-dfad7-5f5d97c8c5-9l9jn"] Jul 06 20:58:00.756196 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:00.756133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-proxy-tls\") pod \"ensemble-graph-305f4-d67d47567-krvxn\" (UID: \"b2eccb3f-95f3-4ffb-aeb4-4604950b7089\") " pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:00.756338 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:58:00.756257 2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-305f4-serving-cert: secret "ensemble-graph-305f4-serving-cert" not found Jul 06 20:58:00.756338 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:58:00.756320 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-proxy-tls podName:b2eccb3f-95f3-4ffb-aeb4-4604950b7089 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:01.756306585 +0000 UTC m=+1372.886669155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-proxy-tls") pod "ensemble-graph-305f4-d67d47567-krvxn" (UID: "b2eccb3f-95f3-4ffb-aeb4-4604950b7089") : secret "ensemble-graph-305f4-serving-cert" not found Jul 06 20:58:01.583814 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:01.583783 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a72cc3a0-b146-4295-add1-c67bdead913a" path="/var/lib/kubelet/pods/a72cc3a0-b146-4295-add1-c67bdead913a/volumes" Jul 06 20:58:01.764555 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:01.764526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-proxy-tls\") pod \"ensemble-graph-305f4-d67d47567-krvxn\" (UID: \"b2eccb3f-95f3-4ffb-aeb4-4604950b7089\") " pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:01.766896 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:01.766875 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-proxy-tls\") pod \"ensemble-graph-305f4-d67d47567-krvxn\" (UID: \"b2eccb3f-95f3-4ffb-aeb4-4604950b7089\") " pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:01.864885 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:01.864867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:01.995002 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:01.994980 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn"] Jul 06 20:58:01.997383 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:58:01.997359 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2eccb3f_95f3_4ffb_aeb4_4604950b7089.slice/crio-5a9a04c055fc942f6ea7a9e4ae4f1a2d8531ecb2bc31b6ab8fcfee6c3e811726 WatchSource:0}: Error finding container 5a9a04c055fc942f6ea7a9e4ae4f1a2d8531ecb2bc31b6ab8fcfee6c3e811726: Status 404 returned error can't find the container with id 5a9a04c055fc942f6ea7a9e4ae4f1a2d8531ecb2bc31b6ab8fcfee6c3e811726 Jul 06 20:58:02.529554 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:02.529520 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" event={"ID":"b2eccb3f-95f3-4ffb-aeb4-4604950b7089","Type":"ContainerStarted","Data":"d6f61f8d3d05cbd7997dc0de04238cd30f65ac168e7e3ea63971ff962fb2cded"} Jul 06 20:58:02.529716 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:02.529576 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" event={"ID":"b2eccb3f-95f3-4ffb-aeb4-4604950b7089","Type":"ContainerStarted","Data":"5a9a04c055fc942f6ea7a9e4ae4f1a2d8531ecb2bc31b6ab8fcfee6c3e811726"} Jul 06 20:58:02.529716 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:02.529634 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:02.548548 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:02.548502 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" podStartSLOduration=2.548487598 podStartE2EDuration="2.548487598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:58:02.547620662 +0000 UTC m=+1373.677983273" watchObservedRunningTime="2026-07-06 20:58:02.548487598 +0000 UTC m=+1373.678850230" Jul 06 20:58:08.539396 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:08.539360 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:10.097988 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:10.097945 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn"] Jul 06 20:58:10.098429 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:10.098126 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" podUID="b2eccb3f-95f3-4ffb-aeb4-4604950b7089" containerName="ensemble-graph-305f4" containerID="cri-o://d6f61f8d3d05cbd7997dc0de04238cd30f65ac168e7e3ea63971ff962fb2cded" gracePeriod=30 Jul 06 20:58:13.537213 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:13.537168 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" podUID="b2eccb3f-95f3-4ffb-aeb4-4604950b7089" containerName="ensemble-graph-305f4" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:58:18.537340 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:18.537305 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" podUID="b2eccb3f-95f3-4ffb-aeb4-4604950b7089" containerName="ensemble-graph-305f4" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:58:23.537502 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:23.537454 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" podUID="b2eccb3f-95f3-4ffb-aeb4-4604950b7089" containerName="ensemble-graph-305f4" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:58:23.538032 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:23.537598 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:28.537498 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:28.537459 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" podUID="b2eccb3f-95f3-4ffb-aeb4-4604950b7089" containerName="ensemble-graph-305f4" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:58:29.712996 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.712948 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v"] Jul 06 20:58:29.713330 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.713215 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a72cc3a0-b146-4295-add1-c67bdead913a" containerName="sequence-graph-dfad7" Jul 06 20:58:29.713330 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.713225 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a72cc3a0-b146-4295-add1-c67bdead913a" containerName="sequence-graph-dfad7" Jul 06 20:58:29.713330 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.713281 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a72cc3a0-b146-4295-add1-c67bdead913a" containerName="sequence-graph-dfad7" Jul 06 20:58:29.717283 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.717265 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" Jul 06 20:58:29.719503 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.719467 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-24ddf-kube-rbac-proxy-sar-config\"" Jul 06 20:58:29.719647 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.719557 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-24ddf-serving-cert\"" Jul 06 20:58:29.727043 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.727021 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v"] Jul 06 20:58:29.849862 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.849834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0b809d-fc36-47f5-b35a-1c5098831d8e-openshift-service-ca-bundle\") pod \"sequence-graph-24ddf-599f97958c-svd5v\" (UID: \"cd0b809d-fc36-47f5-b35a-1c5098831d8e\") " pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" Jul 06 20:58:29.850010 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.849898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd0b809d-fc36-47f5-b35a-1c5098831d8e-proxy-tls\") pod \"sequence-graph-24ddf-599f97958c-svd5v\" (UID: \"cd0b809d-fc36-47f5-b35a-1c5098831d8e\") " pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" Jul 06 20:58:29.950355 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.950328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd0b809d-fc36-47f5-b35a-1c5098831d8e-proxy-tls\") pod \"sequence-graph-24ddf-599f97958c-svd5v\" (UID: \"cd0b809d-fc36-47f5-b35a-1c5098831d8e\") " pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" Jul 06 20:58:29.950463 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.950365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0b809d-fc36-47f5-b35a-1c5098831d8e-openshift-service-ca-bundle\") pod \"sequence-graph-24ddf-599f97958c-svd5v\" (UID: \"cd0b809d-fc36-47f5-b35a-1c5098831d8e\") " pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" Jul 06 20:58:29.950912 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.950893 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0b809d-fc36-47f5-b35a-1c5098831d8e-openshift-service-ca-bundle\") pod \"sequence-graph-24ddf-599f97958c-svd5v\" (UID: \"cd0b809d-fc36-47f5-b35a-1c5098831d8e\") " pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" Jul 06 20:58:29.952597 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:29.952574 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd0b809d-fc36-47f5-b35a-1c5098831d8e-proxy-tls\") pod \"sequence-graph-24ddf-599f97958c-svd5v\" (UID: \"cd0b809d-fc36-47f5-b35a-1c5098831d8e\") " pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" Jul 06 20:58:30.027687 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:30.027635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" Jul 06 20:58:30.149621 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:30.149588 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v"] Jul 06 20:58:30.152254 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:58:30.152228 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0b809d_fc36_47f5_b35a_1c5098831d8e.slice/crio-88f72ac5d4dd356be6975d424293009f7bfa57594312fc2472c078776de2d952 WatchSource:0}: Error finding container 88f72ac5d4dd356be6975d424293009f7bfa57594312fc2472c078776de2d952: Status 404 returned error can't find the container with id 88f72ac5d4dd356be6975d424293009f7bfa57594312fc2472c078776de2d952 Jul 06 20:58:30.633698 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:30.633662 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" event={"ID":"cd0b809d-fc36-47f5-b35a-1c5098831d8e","Type":"ContainerStarted","Data":"228a00beb85a60e60ddefc65f3f6d2ee8c86ab29f85ec662259fab2fcfd2ce06"} Jul 06 20:58:30.633825 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:30.633721 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" event={"ID":"cd0b809d-fc36-47f5-b35a-1c5098831d8e","Type":"ContainerStarted","Data":"88f72ac5d4dd356be6975d424293009f7bfa57594312fc2472c078776de2d952"} Jul 06 20:58:30.633825 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:30.633746 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" Jul 06 20:58:30.652889 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:30.652846 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" podStartSLOduration=1.652832691 podStartE2EDuration="1.652832691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:58:30.651978433 +0000 UTC m=+1401.782341028" watchObservedRunningTime="2026-07-06 20:58:30.652832691 +0000 UTC m=+1401.783195385" Jul 06 20:58:33.537523 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:33.537475 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" podUID="b2eccb3f-95f3-4ffb-aeb4-4604950b7089" containerName="ensemble-graph-305f4" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:58:36.644919 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:36.644887 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" Jul 06 20:58:38.537624 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:38.537574 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" podUID="b2eccb3f-95f3-4ffb-aeb4-4604950b7089" containerName="ensemble-graph-305f4" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:58:39.812009 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:39.811980 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v"] Jul 06 20:58:39.812341 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:39.812140 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" podUID="cd0b809d-fc36-47f5-b35a-1c5098831d8e" containerName="sequence-graph-24ddf" containerID="cri-o://228a00beb85a60e60ddefc65f3f6d2ee8c86ab29f85ec662259fab2fcfd2ce06" gracePeriod=30 Jul 06 20:58:40.241998 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.241976 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:40.418935 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.418867 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-openshift-service-ca-bundle\") pod \"b2eccb3f-95f3-4ffb-aeb4-4604950b7089\" (UID: \"b2eccb3f-95f3-4ffb-aeb4-4604950b7089\") " Jul 06 20:58:40.419071 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.418947 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-proxy-tls\") pod \"b2eccb3f-95f3-4ffb-aeb4-4604950b7089\" (UID: \"b2eccb3f-95f3-4ffb-aeb4-4604950b7089\") " Jul 06 20:58:40.419757 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.419712 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b2eccb3f-95f3-4ffb-aeb4-4604950b7089" (UID: "b2eccb3f-95f3-4ffb-aeb4-4604950b7089"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:58:40.423982 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.421712 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2eccb3f-95f3-4ffb-aeb4-4604950b7089" (UID: "b2eccb3f-95f3-4ffb-aeb4-4604950b7089"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:58:40.520439 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.520414 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-openshift-service-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:58:40.520439 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.520435 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2eccb3f-95f3-4ffb-aeb4-4604950b7089-proxy-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:58:40.671279 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.671209 2579 generic.go:356] "Generic (PLEG): container finished" podID="b2eccb3f-95f3-4ffb-aeb4-4604950b7089" containerID="d6f61f8d3d05cbd7997dc0de04238cd30f65ac168e7e3ea63971ff962fb2cded" exitCode=0 Jul 06 20:58:40.671279 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.671267 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" Jul 06 20:58:40.671279 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.671276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" event={"ID":"b2eccb3f-95f3-4ffb-aeb4-4604950b7089","Type":"ContainerDied","Data":"d6f61f8d3d05cbd7997dc0de04238cd30f65ac168e7e3ea63971ff962fb2cded"} Jul 06 20:58:40.671435 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.671312 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn" event={"ID":"b2eccb3f-95f3-4ffb-aeb4-4604950b7089","Type":"ContainerDied","Data":"5a9a04c055fc942f6ea7a9e4ae4f1a2d8531ecb2bc31b6ab8fcfee6c3e811726"} Jul 06 20:58:40.671435 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.671333 2579 scope.go:117] "RemoveContainer" containerID="d6f61f8d3d05cbd7997dc0de04238cd30f65ac168e7e3ea63971ff962fb2cded" Jul 06 20:58:40.681604 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.681579 2579 scope.go:117] "RemoveContainer" containerID="d6f61f8d3d05cbd7997dc0de04238cd30f65ac168e7e3ea63971ff962fb2cded" Jul 06 20:58:40.682095 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:58:40.682074 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f61f8d3d05cbd7997dc0de04238cd30f65ac168e7e3ea63971ff962fb2cded\": container with ID starting with d6f61f8d3d05cbd7997dc0de04238cd30f65ac168e7e3ea63971ff962fb2cded not found: ID does not exist" containerID="d6f61f8d3d05cbd7997dc0de04238cd30f65ac168e7e3ea63971ff962fb2cded" Jul 06 20:58:40.682202 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.682102 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f61f8d3d05cbd7997dc0de04238cd30f65ac168e7e3ea63971ff962fb2cded"} err="failed to get container status \"d6f61f8d3d05cbd7997dc0de04238cd30f65ac168e7e3ea63971ff962fb2cded\": rpc error: code = NotFound desc = could not find container \"d6f61f8d3d05cbd7997dc0de04238cd30f65ac168e7e3ea63971ff962fb2cded\": container with ID starting with d6f61f8d3d05cbd7997dc0de04238cd30f65ac168e7e3ea63971ff962fb2cded not found: ID does not exist" Jul 06 20:58:40.698097 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.698075 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn"] Jul 06 20:58:40.703920 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:40.703897 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-305f4-d67d47567-krvxn"] Jul 06 20:58:41.583648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:41.583621 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2eccb3f-95f3-4ffb-aeb4-4604950b7089" path="/var/lib/kubelet/pods/b2eccb3f-95f3-4ffb-aeb4-4604950b7089/volumes" Jul 06 20:58:41.642452 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:41.642410 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" podUID="cd0b809d-fc36-47f5-b35a-1c5098831d8e" containerName="sequence-graph-24ddf" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:58:46.642930 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:46.642887 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" podUID="cd0b809d-fc36-47f5-b35a-1c5098831d8e" containerName="sequence-graph-24ddf" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:58:51.641439 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:51.641391 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" podUID="cd0b809d-fc36-47f5-b35a-1c5098831d8e" containerName="sequence-graph-24ddf" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:58:51.641837 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:51.641507 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" Jul 06 20:58:56.642133 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:58:56.642088 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" podUID="cd0b809d-fc36-47f5-b35a-1c5098831d8e" containerName="sequence-graph-24ddf" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:59:01.641523 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:01.641484 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" podUID="cd0b809d-fc36-47f5-b35a-1c5098831d8e" containerName="sequence-graph-24ddf" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:59:06.642037 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:06.641987 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" podUID="cd0b809d-fc36-47f5-b35a-1c5098831d8e" containerName="sequence-graph-24ddf" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:59:09.997545 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:09.997519 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" Jul 06 20:59:10.011669 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.011646 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd0b809d-fc36-47f5-b35a-1c5098831d8e-proxy-tls\") pod \"cd0b809d-fc36-47f5-b35a-1c5098831d8e\" (UID: \"cd0b809d-fc36-47f5-b35a-1c5098831d8e\") " Jul 06 20:59:10.011752 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.011684 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0b809d-fc36-47f5-b35a-1c5098831d8e-openshift-service-ca-bundle\") pod \"cd0b809d-fc36-47f5-b35a-1c5098831d8e\" (UID: \"cd0b809d-fc36-47f5-b35a-1c5098831d8e\") " Jul 06 20:59:10.012048 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.012028 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0b809d-fc36-47f5-b35a-1c5098831d8e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "cd0b809d-fc36-47f5-b35a-1c5098831d8e" (UID: "cd0b809d-fc36-47f5-b35a-1c5098831d8e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:59:10.013423 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.013405 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0b809d-fc36-47f5-b35a-1c5098831d8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd0b809d-fc36-47f5-b35a-1c5098831d8e" (UID: "cd0b809d-fc36-47f5-b35a-1c5098831d8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:59:10.112116 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.112096 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0b809d-fc36-47f5-b35a-1c5098831d8e-openshift-service-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:59:10.112116 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.112117 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd0b809d-fc36-47f5-b35a-1c5098831d8e-proxy-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 20:59:10.352173 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.352143 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h"] Jul 06 20:59:10.352412 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.352400 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd0b809d-fc36-47f5-b35a-1c5098831d8e" containerName="sequence-graph-24ddf" Jul 06 20:59:10.352454 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.352414 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd0b809d-fc36-47f5-b35a-1c5098831d8e" containerName="sequence-graph-24ddf" Jul 06 20:59:10.352454 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.352429 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2eccb3f-95f3-4ffb-aeb4-4604950b7089" containerName="ensemble-graph-305f4" Jul 06 20:59:10.352454 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.352434 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2eccb3f-95f3-4ffb-aeb4-4604950b7089" containerName="ensemble-graph-305f4" Jul 06 20:59:10.352537 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.352479 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2eccb3f-95f3-4ffb-aeb4-4604950b7089" containerName="ensemble-graph-305f4" Jul 06 20:59:10.352537 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.352488 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd0b809d-fc36-47f5-b35a-1c5098831d8e" containerName="sequence-graph-24ddf" Jul 06 20:59:10.355266 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.355248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 20:59:10.357191 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.357171 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-858c3-kube-rbac-proxy-sar-config\"" Jul 06 20:59:10.357533 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.357512 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-858c3-serving-cert\"" Jul 06 20:59:10.366603 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.366558 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h"] Jul 06 20:59:10.414159 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.414118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-proxy-tls\") pod \"ensemble-graph-858c3-7798d67676-c5m7h\" (UID: \"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828\") " pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 20:59:10.414297 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.414225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-openshift-service-ca-bundle\") pod \"ensemble-graph-858c3-7798d67676-c5m7h\" (UID: \"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828\") " pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 20:59:10.514667 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.514640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-proxy-tls\") pod \"ensemble-graph-858c3-7798d67676-c5m7h\" (UID: \"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828\") " pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 20:59:10.514810 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.514684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-openshift-service-ca-bundle\") pod \"ensemble-graph-858c3-7798d67676-c5m7h\" (UID: \"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828\") " pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 20:59:10.514872 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:59:10.514806 2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-858c3-serving-cert: secret "ensemble-graph-858c3-serving-cert" not found Jul 06 20:59:10.514930 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:59:10.514874 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-proxy-tls podName:f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:11.01485286 +0000 UTC m=+1442.145215445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-proxy-tls") pod "ensemble-graph-858c3-7798d67676-c5m7h" (UID: "f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828") : secret "ensemble-graph-858c3-serving-cert" not found Jul 06 20:59:10.515285 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.515269 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-openshift-service-ca-bundle\") pod \"ensemble-graph-858c3-7798d67676-c5m7h\" (UID: \"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828\") " pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 20:59:10.780070 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.780001 2579 generic.go:356] "Generic (PLEG): container finished" podID="cd0b809d-fc36-47f5-b35a-1c5098831d8e" containerID="228a00beb85a60e60ddefc65f3f6d2ee8c86ab29f85ec662259fab2fcfd2ce06" exitCode=137 Jul 06 20:59:10.780070 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.780052 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" event={"ID":"cd0b809d-fc36-47f5-b35a-1c5098831d8e","Type":"ContainerDied","Data":"228a00beb85a60e60ddefc65f3f6d2ee8c86ab29f85ec662259fab2fcfd2ce06"} Jul 06 20:59:10.780070 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.780054 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" Jul 06 20:59:10.780263 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.780093 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v" event={"ID":"cd0b809d-fc36-47f5-b35a-1c5098831d8e","Type":"ContainerDied","Data":"88f72ac5d4dd356be6975d424293009f7bfa57594312fc2472c078776de2d952"} Jul 06 20:59:10.780263 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.780116 2579 scope.go:117] "RemoveContainer" containerID="228a00beb85a60e60ddefc65f3f6d2ee8c86ab29f85ec662259fab2fcfd2ce06" Jul 06 20:59:10.791705 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.791682 2579 scope.go:117] "RemoveContainer" containerID="228a00beb85a60e60ddefc65f3f6d2ee8c86ab29f85ec662259fab2fcfd2ce06" Jul 06 20:59:10.792125 ip-10-0-132-244 kubenswrapper[2579]: E0706 20:59:10.792006 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228a00beb85a60e60ddefc65f3f6d2ee8c86ab29f85ec662259fab2fcfd2ce06\": container with ID starting with 228a00beb85a60e60ddefc65f3f6d2ee8c86ab29f85ec662259fab2fcfd2ce06 not found: ID does not exist" containerID="228a00beb85a60e60ddefc65f3f6d2ee8c86ab29f85ec662259fab2fcfd2ce06" Jul 06 20:59:10.792125 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.792055 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228a00beb85a60e60ddefc65f3f6d2ee8c86ab29f85ec662259fab2fcfd2ce06"} err="failed to get container status \"228a00beb85a60e60ddefc65f3f6d2ee8c86ab29f85ec662259fab2fcfd2ce06\": rpc error: code = NotFound desc = could not find container \"228a00beb85a60e60ddefc65f3f6d2ee8c86ab29f85ec662259fab2fcfd2ce06\": container with ID starting with 228a00beb85a60e60ddefc65f3f6d2ee8c86ab29f85ec662259fab2fcfd2ce06 not found: ID does not exist" Jul 06 20:59:10.813676 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.813646 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v"] Jul 06 20:59:10.816317 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:10.816293 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-24ddf-599f97958c-svd5v"] Jul 06 20:59:11.018819 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:11.018765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-proxy-tls\") pod \"ensemble-graph-858c3-7798d67676-c5m7h\" (UID: \"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828\") " pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 20:59:11.021241 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:11.021215 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-proxy-tls\") pod \"ensemble-graph-858c3-7798d67676-c5m7h\" (UID: \"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828\") " pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 20:59:11.264365 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:11.264335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 20:59:11.398648 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:11.398619 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h"] Jul 06 20:59:11.400915 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:59:11.400890 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d1ae92_8e3f_4906_bdd7_f5a94ed1e828.slice/crio-c7aefd48f78d05d83d82bc13278825aae14e3d363de732d01bd86eb1df203cae WatchSource:0}: Error finding container c7aefd48f78d05d83d82bc13278825aae14e3d363de732d01bd86eb1df203cae: Status 404 returned error can't find the container with id c7aefd48f78d05d83d82bc13278825aae14e3d363de732d01bd86eb1df203cae Jul 06 20:59:11.583498 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:11.583433 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd0b809d-fc36-47f5-b35a-1c5098831d8e" path="/var/lib/kubelet/pods/cd0b809d-fc36-47f5-b35a-1c5098831d8e/volumes" Jul 06 20:59:11.784946 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:11.784914 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" event={"ID":"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828","Type":"ContainerStarted","Data":"1ff162d3aff4ebba3a837cf3c527219ed48b8a1e91144ac815710f8b2d3f4c3a"} Jul 06 20:59:11.785106 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:11.784978 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" event={"ID":"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828","Type":"ContainerStarted","Data":"c7aefd48f78d05d83d82bc13278825aae14e3d363de732d01bd86eb1df203cae"} Jul 06 20:59:11.785106 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:11.785032 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 20:59:11.802854 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:11.802809 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" podStartSLOduration=1.802798105 podStartE2EDuration="1.802798105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:11.802475731 +0000 UTC m=+1442.932838325" watchObservedRunningTime="2026-07-06 20:59:11.802798105 +0000 UTC m=+1442.933160702" Jul 06 20:59:17.795623 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:17.795594 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 20:59:40.060475 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.060444 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc"] Jul 06 20:59:40.063641 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.063619 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" Jul 06 20:59:40.065657 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.065636 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-92a61-kube-rbac-proxy-sar-config\"" Jul 06 20:59:40.065657 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.065653 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-92a61-serving-cert\"" Jul 06 20:59:40.072242 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.072219 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc"] Jul 06 20:59:40.120682 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.120662 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d985656-6014-4421-86e4-dcf731ef19e3-proxy-tls\") pod \"sequence-graph-92a61-78458fdc5-fngpc\" (UID: \"3d985656-6014-4421-86e4-dcf731ef19e3\") " pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" Jul 06 20:59:40.120773 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.120689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d985656-6014-4421-86e4-dcf731ef19e3-openshift-service-ca-bundle\") pod \"sequence-graph-92a61-78458fdc5-fngpc\" (UID: \"3d985656-6014-4421-86e4-dcf731ef19e3\") " pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" Jul 06 20:59:40.221028 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.221000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d985656-6014-4421-86e4-dcf731ef19e3-proxy-tls\") pod \"sequence-graph-92a61-78458fdc5-fngpc\" (UID: \"3d985656-6014-4421-86e4-dcf731ef19e3\") " pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" Jul 06 20:59:40.221155 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.221032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d985656-6014-4421-86e4-dcf731ef19e3-openshift-service-ca-bundle\") pod \"sequence-graph-92a61-78458fdc5-fngpc\" (UID: \"3d985656-6014-4421-86e4-dcf731ef19e3\") " pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" Jul 06 20:59:40.221553 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.221537 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d985656-6014-4421-86e4-dcf731ef19e3-openshift-service-ca-bundle\") pod \"sequence-graph-92a61-78458fdc5-fngpc\" (UID: \"3d985656-6014-4421-86e4-dcf731ef19e3\") " pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" Jul 06 20:59:40.223300 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.223276 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d985656-6014-4421-86e4-dcf731ef19e3-proxy-tls\") pod \"sequence-graph-92a61-78458fdc5-fngpc\" (UID: \"3d985656-6014-4421-86e4-dcf731ef19e3\") " pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" Jul 06 20:59:40.374688 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.374664 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" Jul 06 20:59:40.510386 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.510357 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc"] Jul 06 20:59:40.511946 ip-10-0-132-244 kubenswrapper[2579]: W0706 20:59:40.511917 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d985656_6014_4421_86e4_dcf731ef19e3.slice/crio-b79c33f6f9f50c7e415c5ea486a2f21921bdcab09819f38f38e4871ce37a96ff WatchSource:0}: Error finding container b79c33f6f9f50c7e415c5ea486a2f21921bdcab09819f38f38e4871ce37a96ff: Status 404 returned error can't find the container with id b79c33f6f9f50c7e415c5ea486a2f21921bdcab09819f38f38e4871ce37a96ff Jul 06 20:59:40.889111 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.889070 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" event={"ID":"3d985656-6014-4421-86e4-dcf731ef19e3","Type":"ContainerStarted","Data":"71d67d8fff08ca349661e493c8dd14440f68368c958c63e8963ca330296a0a18"} Jul 06 20:59:40.889243 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.889140 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" event={"ID":"3d985656-6014-4421-86e4-dcf731ef19e3","Type":"ContainerStarted","Data":"b79c33f6f9f50c7e415c5ea486a2f21921bdcab09819f38f38e4871ce37a96ff"} Jul 06 20:59:40.889243 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.889202 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" Jul 06 20:59:40.908461 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:40.908411 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" podStartSLOduration=0.90839639 podStartE2EDuration="908.39639ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:40.907801978 +0000 UTC m=+1472.038164570" watchObservedRunningTime="2026-07-06 20:59:40.90839639 +0000 UTC m=+1472.038759118" Jul 06 20:59:46.898869 ip-10-0-132-244 kubenswrapper[2579]: I0706 20:59:46.898836 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" Jul 06 21:00:09.387675 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:00:09.387553 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 21:00:09.401066 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:00:09.390276 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 21:05:09.409869 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:05:09.409754 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 21:05:09.414832 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:05:09.412878 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 21:07:24.873906 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:24.873872 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h"] Jul 06 21:07:24.874401 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:24.874130 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" podUID="f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" containerName="ensemble-graph-858c3" containerID="cri-o://1ff162d3aff4ebba3a837cf3c527219ed48b8a1e91144ac815710f8b2d3f4c3a" gracePeriod=30 Jul 06 21:07:27.794514 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:27.794478 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" podUID="f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" containerName="ensemble-graph-858c3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:07:32.794458 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:32.794419 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" podUID="f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" containerName="ensemble-graph-858c3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:07:37.794048 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:37.794011 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" podUID="f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" containerName="ensemble-graph-858c3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:07:37.794400 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:37.794120 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 21:07:42.794034 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:42.793994 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" podUID="f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" containerName="ensemble-graph-858c3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:07:47.793753 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:47.793711 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" podUID="f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" containerName="ensemble-graph-858c3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:07:52.794267 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:52.794223 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" podUID="f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" containerName="ensemble-graph-858c3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:07:54.741882 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:54.741852 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc"] Jul 06 21:07:54.742285 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:54.742040 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" podUID="3d985656-6014-4421-86e4-dcf731ef19e3" containerName="sequence-graph-92a61" containerID="cri-o://71d67d8fff08ca349661e493c8dd14440f68368c958c63e8963ca330296a0a18" gracePeriod=30 Jul 06 21:07:55.032835 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.032813 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 21:07:55.118356 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.118323 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-proxy-tls\") pod \"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828\" (UID: \"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828\") " Jul 06 21:07:55.118500 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.118363 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-openshift-service-ca-bundle\") pod \"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828\" (UID: \"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828\") " Jul 06 21:07:55.118757 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.118725 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" (UID: "f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:07:55.120626 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.120601 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" (UID: "f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:07:55.219263 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.219230 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-proxy-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 21:07:55.219263 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.219257 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828-openshift-service-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 21:07:55.644159 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.644126 2579 generic.go:356] "Generic (PLEG): container finished" podID="f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" containerID="1ff162d3aff4ebba3a837cf3c527219ed48b8a1e91144ac815710f8b2d3f4c3a" exitCode=0 Jul 06 21:07:55.644298 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.644172 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" Jul 06 21:07:55.644298 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.644196 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" event={"ID":"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828","Type":"ContainerDied","Data":"1ff162d3aff4ebba3a837cf3c527219ed48b8a1e91144ac815710f8b2d3f4c3a"} Jul 06 21:07:55.644298 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.644236 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h" event={"ID":"f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828","Type":"ContainerDied","Data":"c7aefd48f78d05d83d82bc13278825aae14e3d363de732d01bd86eb1df203cae"} Jul 06 21:07:55.644298 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.644262 2579 scope.go:117] "RemoveContainer" containerID="1ff162d3aff4ebba3a837cf3c527219ed48b8a1e91144ac815710f8b2d3f4c3a" Jul 06 21:07:55.653696 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.653677 2579 scope.go:117] "RemoveContainer" containerID="1ff162d3aff4ebba3a837cf3c527219ed48b8a1e91144ac815710f8b2d3f4c3a" Jul 06 21:07:55.654078 ip-10-0-132-244 kubenswrapper[2579]: E0706 21:07:55.654050 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff162d3aff4ebba3a837cf3c527219ed48b8a1e91144ac815710f8b2d3f4c3a\": container with ID starting with 1ff162d3aff4ebba3a837cf3c527219ed48b8a1e91144ac815710f8b2d3f4c3a not found: ID does not exist" containerID="1ff162d3aff4ebba3a837cf3c527219ed48b8a1e91144ac815710f8b2d3f4c3a" Jul 06 21:07:55.654187 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.654087 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff162d3aff4ebba3a837cf3c527219ed48b8a1e91144ac815710f8b2d3f4c3a"} err="failed to get container status \"1ff162d3aff4ebba3a837cf3c527219ed48b8a1e91144ac815710f8b2d3f4c3a\": rpc error: code = NotFound desc = could not find container \"1ff162d3aff4ebba3a837cf3c527219ed48b8a1e91144ac815710f8b2d3f4c3a\": container with ID starting with 1ff162d3aff4ebba3a837cf3c527219ed48b8a1e91144ac815710f8b2d3f4c3a not found: ID does not exist" Jul 06 21:07:55.675323 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.675299 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h"] Jul 06 21:07:55.682406 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:55.682387 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-858c3-7798d67676-c5m7h"] Jul 06 21:07:56.897421 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:56.897377 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" podUID="3d985656-6014-4421-86e4-dcf731ef19e3" containerName="sequence-graph-92a61" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:07:57.583672 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:07:57.583639 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" path="/var/lib/kubelet/pods/f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828/volumes" Jul 06 21:08:01.897563 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:01.897525 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" podUID="3d985656-6014-4421-86e4-dcf731ef19e3" containerName="sequence-graph-92a61" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:08:06.897783 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:06.897742 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" podUID="3d985656-6014-4421-86e4-dcf731ef19e3" containerName="sequence-graph-92a61" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:08:06.898157 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:06.897848 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" Jul 06 21:08:11.897796 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:11.897759 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" podUID="3d985656-6014-4421-86e4-dcf731ef19e3" containerName="sequence-graph-92a61" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:08:16.897163 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:16.897128 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" podUID="3d985656-6014-4421-86e4-dcf731ef19e3" containerName="sequence-graph-92a61" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:08:21.897695 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:21.897658 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" podUID="3d985656-6014-4421-86e4-dcf731ef19e3" containerName="sequence-graph-92a61" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:08:24.877114 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:24.877092 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" Jul 06 21:08:24.918763 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:24.918740 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d985656-6014-4421-86e4-dcf731ef19e3-proxy-tls\") pod \"3d985656-6014-4421-86e4-dcf731ef19e3\" (UID: \"3d985656-6014-4421-86e4-dcf731ef19e3\") " Jul 06 21:08:24.918900 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:24.918800 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d985656-6014-4421-86e4-dcf731ef19e3-openshift-service-ca-bundle\") pod \"3d985656-6014-4421-86e4-dcf731ef19e3\" (UID: \"3d985656-6014-4421-86e4-dcf731ef19e3\") " Jul 06 21:08:24.919191 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:24.919163 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d985656-6014-4421-86e4-dcf731ef19e3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3d985656-6014-4421-86e4-dcf731ef19e3" (UID: "3d985656-6014-4421-86e4-dcf731ef19e3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:08:24.920706 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:24.920677 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d985656-6014-4421-86e4-dcf731ef19e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d985656-6014-4421-86e4-dcf731ef19e3" (UID: "3d985656-6014-4421-86e4-dcf731ef19e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:08:25.020251 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.020188 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d985656-6014-4421-86e4-dcf731ef19e3-proxy-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 21:08:25.020251 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.020212 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d985656-6014-4421-86e4-dcf731ef19e3-openshift-service-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 21:08:25.138985 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.138941 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp"] Jul 06 21:08:25.139224 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.139211 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d985656-6014-4421-86e4-dcf731ef19e3" containerName="sequence-graph-92a61" Jul 06 21:08:25.139277 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.139225 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d985656-6014-4421-86e4-dcf731ef19e3" containerName="sequence-graph-92a61" Jul 06 21:08:25.139277 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.139240 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" containerName="ensemble-graph-858c3" Jul 06 21:08:25.139277 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.139245 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" containerName="ensemble-graph-858c3" Jul 06 21:08:25.139368 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.139297 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2d1ae92-8e3f-4906-bdd7-f5a94ed1e828" containerName="ensemble-graph-858c3" Jul 06 21:08:25.139368 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.139306 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d985656-6014-4421-86e4-dcf731ef19e3" containerName="sequence-graph-92a61" Jul 06 21:08:25.142063 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.142047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:08:25.144408 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.144388 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-999e1-kube-rbac-proxy-sar-config\"" Jul 06 21:08:25.144881 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.144867 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-999e1-serving-cert\"" Jul 06 21:08:25.153524 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.153502 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp"] Jul 06 21:08:25.222322 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.222283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db11150-7f15-404d-a5a8-d63156df21a1-proxy-tls\") pod \"splitter-graph-999e1-5df54fbb4c-8s8kp\" (UID: \"4db11150-7f15-404d-a5a8-d63156df21a1\") " pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:08:25.222511 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.222390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4db11150-7f15-404d-a5a8-d63156df21a1-openshift-service-ca-bundle\") pod \"splitter-graph-999e1-5df54fbb4c-8s8kp\" (UID: \"4db11150-7f15-404d-a5a8-d63156df21a1\") " pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:08:25.323837 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.323762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db11150-7f15-404d-a5a8-d63156df21a1-proxy-tls\") pod \"splitter-graph-999e1-5df54fbb4c-8s8kp\" (UID: \"4db11150-7f15-404d-a5a8-d63156df21a1\") " pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:08:25.323837 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.323805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4db11150-7f15-404d-a5a8-d63156df21a1-openshift-service-ca-bundle\") pod \"splitter-graph-999e1-5df54fbb4c-8s8kp\" (UID: \"4db11150-7f15-404d-a5a8-d63156df21a1\") " pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:08:25.324028 ip-10-0-132-244 kubenswrapper[2579]: E0706 21:08:25.323906 2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-999e1-serving-cert: secret "splitter-graph-999e1-serving-cert" not found Jul 06 21:08:25.324028 ip-10-0-132-244 kubenswrapper[2579]: E0706 21:08:25.323989 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4db11150-7f15-404d-a5a8-d63156df21a1-proxy-tls podName:4db11150-7f15-404d-a5a8-d63156df21a1 nodeName:}" failed. No retries permitted until 2026-07-06 21:08:25.823972792 +0000 UTC m=+1996.954335377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4db11150-7f15-404d-a5a8-d63156df21a1-proxy-tls") pod "splitter-graph-999e1-5df54fbb4c-8s8kp" (UID: "4db11150-7f15-404d-a5a8-d63156df21a1") : secret "splitter-graph-999e1-serving-cert" not found Jul 06 21:08:25.324411 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.324394 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4db11150-7f15-404d-a5a8-d63156df21a1-openshift-service-ca-bundle\") pod \"splitter-graph-999e1-5df54fbb4c-8s8kp\" (UID: \"4db11150-7f15-404d-a5a8-d63156df21a1\") " pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:08:25.745617 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.745584 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d985656-6014-4421-86e4-dcf731ef19e3" containerID="71d67d8fff08ca349661e493c8dd14440f68368c958c63e8963ca330296a0a18" exitCode=0 Jul 06 21:08:25.745767 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.745622 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" event={"ID":"3d985656-6014-4421-86e4-dcf731ef19e3","Type":"ContainerDied","Data":"71d67d8fff08ca349661e493c8dd14440f68368c958c63e8963ca330296a0a18"} Jul 06 21:08:25.745767 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.745628 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" Jul 06 21:08:25.745767 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.745662 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc" event={"ID":"3d985656-6014-4421-86e4-dcf731ef19e3","Type":"ContainerDied","Data":"b79c33f6f9f50c7e415c5ea486a2f21921bdcab09819f38f38e4871ce37a96ff"} Jul 06 21:08:25.745767 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.745684 2579 scope.go:117] "RemoveContainer" containerID="71d67d8fff08ca349661e493c8dd14440f68368c958c63e8963ca330296a0a18" Jul 06 21:08:25.754290 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.754265 2579 scope.go:117] "RemoveContainer" containerID="71d67d8fff08ca349661e493c8dd14440f68368c958c63e8963ca330296a0a18" Jul 06 21:08:25.754549 ip-10-0-132-244 kubenswrapper[2579]: E0706 21:08:25.754532 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d67d8fff08ca349661e493c8dd14440f68368c958c63e8963ca330296a0a18\": container with ID starting with 71d67d8fff08ca349661e493c8dd14440f68368c958c63e8963ca330296a0a18 not found: ID does not exist" containerID="71d67d8fff08ca349661e493c8dd14440f68368c958c63e8963ca330296a0a18" Jul 06 21:08:25.754639 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.754557 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d67d8fff08ca349661e493c8dd14440f68368c958c63e8963ca330296a0a18"} err="failed to get container status \"71d67d8fff08ca349661e493c8dd14440f68368c958c63e8963ca330296a0a18\": rpc error: code = NotFound desc = could not find container \"71d67d8fff08ca349661e493c8dd14440f68368c958c63e8963ca330296a0a18\": container with ID starting with 71d67d8fff08ca349661e493c8dd14440f68368c958c63e8963ca330296a0a18 not found: ID does not exist" Jul 06 21:08:25.763354 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.763333 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc"] Jul 06 21:08:25.768293 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.768273 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-92a61-78458fdc5-fngpc"] Jul 06 21:08:25.828450 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.828421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db11150-7f15-404d-a5a8-d63156df21a1-proxy-tls\") pod \"splitter-graph-999e1-5df54fbb4c-8s8kp\" (UID: \"4db11150-7f15-404d-a5a8-d63156df21a1\") " pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:08:25.833394 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:25.831750 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db11150-7f15-404d-a5a8-d63156df21a1-proxy-tls\") pod \"splitter-graph-999e1-5df54fbb4c-8s8kp\" (UID: \"4db11150-7f15-404d-a5a8-d63156df21a1\") " pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:08:26.051731 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:26.051698 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:08:26.178945 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:26.178918 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp"] Jul 06 21:08:26.181171 ip-10-0-132-244 kubenswrapper[2579]: W0706 21:08:26.181146 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db11150_7f15_404d_a5a8_d63156df21a1.slice/crio-63e2e0fed74a50b57ab6226d65342c094fdf8e7dd44f9ead2349d07b5f3fb7e5 WatchSource:0}: Error finding container 63e2e0fed74a50b57ab6226d65342c094fdf8e7dd44f9ead2349d07b5f3fb7e5: Status 404 returned error can't find the container with id 63e2e0fed74a50b57ab6226d65342c094fdf8e7dd44f9ead2349d07b5f3fb7e5 Jul 06 21:08:26.751217 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:26.751180 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" event={"ID":"4db11150-7f15-404d-a5a8-d63156df21a1","Type":"ContainerStarted","Data":"d592363b8362bed89b75c50b35e10a3a43bf7ac84b5deb638d37e7e538ebdea3"} Jul 06 21:08:26.751408 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:26.751227 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" event={"ID":"4db11150-7f15-404d-a5a8-d63156df21a1","Type":"ContainerStarted","Data":"63e2e0fed74a50b57ab6226d65342c094fdf8e7dd44f9ead2349d07b5f3fb7e5"} Jul 06 21:08:26.751408 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:26.751268 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:08:26.770570 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:26.770511 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" podStartSLOduration=1.77049359 podStartE2EDuration="1.77049359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:26.770093173 +0000 UTC m=+1997.900455767" watchObservedRunningTime="2026-07-06 21:08:26.77049359 +0000 UTC m=+1997.900856183" Jul 06 21:08:27.583832 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:27.583801 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d985656-6014-4421-86e4-dcf731ef19e3" path="/var/lib/kubelet/pods/3d985656-6014-4421-86e4-dcf731ef19e3/volumes" Jul 06 21:08:32.761379 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:32.761341 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:08:35.231508 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:35.231477 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp"] Jul 06 21:08:35.231857 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:35.231643 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" podUID="4db11150-7f15-404d-a5a8-d63156df21a1" containerName="splitter-graph-999e1" containerID="cri-o://d592363b8362bed89b75c50b35e10a3a43bf7ac84b5deb638d37e7e538ebdea3" gracePeriod=30 Jul 06 21:08:37.760019 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:37.759978 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" podUID="4db11150-7f15-404d-a5a8-d63156df21a1" containerName="splitter-graph-999e1" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:08:42.759666 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:42.759620 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" podUID="4db11150-7f15-404d-a5a8-d63156df21a1" containerName="splitter-graph-999e1" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:08:47.760263 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:47.760221 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" podUID="4db11150-7f15-404d-a5a8-d63156df21a1" containerName="splitter-graph-999e1" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:08:47.760615 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:47.760324 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:08:52.759843 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:52.759807 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" podUID="4db11150-7f15-404d-a5a8-d63156df21a1" containerName="splitter-graph-999e1" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:08:54.986877 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:54.986844 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9"] Jul 06 21:08:54.991413 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:54.991378 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:08:54.994736 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:54.994714 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-32e32-serving-cert\"" Jul 06 21:08:54.994860 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:54.994717 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-32e32-kube-rbac-proxy-sar-config\"" Jul 06 21:08:55.002468 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:55.002445 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9"] Jul 06 21:08:55.137915 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:55.137879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa8cd06-104b-485e-897d-850fc8300810-proxy-tls\") pod \"switch-graph-32e32-56c9887cf-bgkh9\" (UID: \"0fa8cd06-104b-485e-897d-850fc8300810\") " pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:08:55.138082 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:55.137920 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa8cd06-104b-485e-897d-850fc8300810-openshift-service-ca-bundle\") pod \"switch-graph-32e32-56c9887cf-bgkh9\" (UID: \"0fa8cd06-104b-485e-897d-850fc8300810\") " pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:08:55.238448 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:55.238375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa8cd06-104b-485e-897d-850fc8300810-proxy-tls\") pod \"switch-graph-32e32-56c9887cf-bgkh9\" (UID: \"0fa8cd06-104b-485e-897d-850fc8300810\") " pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:08:55.238448 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:55.238414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa8cd06-104b-485e-897d-850fc8300810-openshift-service-ca-bundle\") pod \"switch-graph-32e32-56c9887cf-bgkh9\" (UID: \"0fa8cd06-104b-485e-897d-850fc8300810\") " pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:08:55.238614 ip-10-0-132-244 kubenswrapper[2579]: E0706 21:08:55.238498 2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-32e32-serving-cert: secret "switch-graph-32e32-serving-cert" not found Jul 06 21:08:55.238614 ip-10-0-132-244 kubenswrapper[2579]: E0706 21:08:55.238567 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa8cd06-104b-485e-897d-850fc8300810-proxy-tls podName:0fa8cd06-104b-485e-897d-850fc8300810 nodeName:}" failed. No retries permitted until 2026-07-06 21:08:55.738551207 +0000 UTC m=+2026.868913779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0fa8cd06-104b-485e-897d-850fc8300810-proxy-tls") pod "switch-graph-32e32-56c9887cf-bgkh9" (UID: "0fa8cd06-104b-485e-897d-850fc8300810") : secret "switch-graph-32e32-serving-cert" not found Jul 06 21:08:55.238989 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:55.238974 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa8cd06-104b-485e-897d-850fc8300810-openshift-service-ca-bundle\") pod \"switch-graph-32e32-56c9887cf-bgkh9\" (UID: \"0fa8cd06-104b-485e-897d-850fc8300810\") " pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:08:55.743198 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:55.743169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa8cd06-104b-485e-897d-850fc8300810-proxy-tls\") pod \"switch-graph-32e32-56c9887cf-bgkh9\" (UID: \"0fa8cd06-104b-485e-897d-850fc8300810\") " pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:08:55.745443 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:55.745420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa8cd06-104b-485e-897d-850fc8300810-proxy-tls\") pod \"switch-graph-32e32-56c9887cf-bgkh9\" (UID: \"0fa8cd06-104b-485e-897d-850fc8300810\") " pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:08:55.901606 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:55.901566 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:08:56.031671 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:56.031588 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9"] Jul 06 21:08:56.034989 ip-10-0-132-244 kubenswrapper[2579]: W0706 21:08:56.034939 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa8cd06_104b_485e_897d_850fc8300810.slice/crio-ee3c787ccd65d85d524f9f685e6e0d537dbc207884356cf0f480b673afd1eedc WatchSource:0}: Error finding container ee3c787ccd65d85d524f9f685e6e0d537dbc207884356cf0f480b673afd1eedc: Status 404 returned error can't find the container with id ee3c787ccd65d85d524f9f685e6e0d537dbc207884356cf0f480b673afd1eedc Jul 06 21:08:56.856433 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:56.856394 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" event={"ID":"0fa8cd06-104b-485e-897d-850fc8300810","Type":"ContainerStarted","Data":"74ccb6930e7f90f98d865a68ec00d9dafe5968fc995eb4a6081d91572db694c0"} Jul 06 21:08:56.856590 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:56.856453 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" event={"ID":"0fa8cd06-104b-485e-897d-850fc8300810","Type":"ContainerStarted","Data":"ee3c787ccd65d85d524f9f685e6e0d537dbc207884356cf0f480b673afd1eedc"} Jul 06 21:08:56.856590 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:56.856520 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:08:56.876523 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:56.876475 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" podStartSLOduration=2.876461548 podStartE2EDuration="2.876461548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:56.875364606 +0000 UTC m=+2028.005727199" watchObservedRunningTime="2026-07-06 21:08:56.876461548 +0000 UTC m=+2028.006824184" Jul 06 21:08:57.759036 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:08:57.758999 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" podUID="4db11150-7f15-404d-a5a8-d63156df21a1" containerName="splitter-graph-999e1" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:09:02.759794 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:02.759749 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" podUID="4db11150-7f15-404d-a5a8-d63156df21a1" containerName="splitter-graph-999e1" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:09:02.866037 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:02.866010 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:09:05.373615 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.373592 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:09:05.509818 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.509749 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4db11150-7f15-404d-a5a8-d63156df21a1-openshift-service-ca-bundle\") pod \"4db11150-7f15-404d-a5a8-d63156df21a1\" (UID: \"4db11150-7f15-404d-a5a8-d63156df21a1\") " Jul 06 21:09:05.509818 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.509807 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db11150-7f15-404d-a5a8-d63156df21a1-proxy-tls\") pod \"4db11150-7f15-404d-a5a8-d63156df21a1\" (UID: \"4db11150-7f15-404d-a5a8-d63156df21a1\") " Jul 06 21:09:05.510122 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.510100 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db11150-7f15-404d-a5a8-d63156df21a1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4db11150-7f15-404d-a5a8-d63156df21a1" (UID: "4db11150-7f15-404d-a5a8-d63156df21a1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:09:05.511722 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.511701 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db11150-7f15-404d-a5a8-d63156df21a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4db11150-7f15-404d-a5a8-d63156df21a1" (UID: "4db11150-7f15-404d-a5a8-d63156df21a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:09:05.610725 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.610705 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4db11150-7f15-404d-a5a8-d63156df21a1-openshift-service-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 21:09:05.610824 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.610728 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db11150-7f15-404d-a5a8-d63156df21a1-proxy-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 21:09:05.890491 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.890458 2579 generic.go:356] "Generic (PLEG): container finished" podID="4db11150-7f15-404d-a5a8-d63156df21a1" containerID="d592363b8362bed89b75c50b35e10a3a43bf7ac84b5deb638d37e7e538ebdea3" exitCode=0 Jul 06 21:09:05.890624 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.890494 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" event={"ID":"4db11150-7f15-404d-a5a8-d63156df21a1","Type":"ContainerDied","Data":"d592363b8362bed89b75c50b35e10a3a43bf7ac84b5deb638d37e7e538ebdea3"} Jul 06 21:09:05.890624 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.890510 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" Jul 06 21:09:05.890624 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.890548 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp" event={"ID":"4db11150-7f15-404d-a5a8-d63156df21a1","Type":"ContainerDied","Data":"63e2e0fed74a50b57ab6226d65342c094fdf8e7dd44f9ead2349d07b5f3fb7e5"} Jul 06 21:09:05.890624 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.890575 2579 scope.go:117] "RemoveContainer" containerID="d592363b8362bed89b75c50b35e10a3a43bf7ac84b5deb638d37e7e538ebdea3" Jul 06 21:09:05.898360 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.898340 2579 scope.go:117] "RemoveContainer" containerID="d592363b8362bed89b75c50b35e10a3a43bf7ac84b5deb638d37e7e538ebdea3" Jul 06 21:09:05.898614 ip-10-0-132-244 kubenswrapper[2579]: E0706 21:09:05.898596 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d592363b8362bed89b75c50b35e10a3a43bf7ac84b5deb638d37e7e538ebdea3\": container with ID starting with d592363b8362bed89b75c50b35e10a3a43bf7ac84b5deb638d37e7e538ebdea3 not found: ID does not exist" containerID="d592363b8362bed89b75c50b35e10a3a43bf7ac84b5deb638d37e7e538ebdea3" Jul 06 21:09:05.898682 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.898628 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d592363b8362bed89b75c50b35e10a3a43bf7ac84b5deb638d37e7e538ebdea3"} err="failed to get container status \"d592363b8362bed89b75c50b35e10a3a43bf7ac84b5deb638d37e7e538ebdea3\": rpc error: code = NotFound desc = could not find container \"d592363b8362bed89b75c50b35e10a3a43bf7ac84b5deb638d37e7e538ebdea3\": container with ID starting with d592363b8362bed89b75c50b35e10a3a43bf7ac84b5deb638d37e7e538ebdea3 not found: ID does not exist" Jul 06 21:09:05.909551 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.909530 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp"] Jul 06 21:09:05.918329 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:05.918310 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-999e1-5df54fbb4c-8s8kp"] Jul 06 21:09:07.583297 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:07.583264 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4db11150-7f15-404d-a5a8-d63156df21a1" path="/var/lib/kubelet/pods/4db11150-7f15-404d-a5a8-d63156df21a1/volumes" Jul 06 21:09:35.471228 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.471193 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z"] Jul 06 21:09:35.471578 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.471467 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db11150-7f15-404d-a5a8-d63156df21a1" containerName="splitter-graph-999e1" Jul 06 21:09:35.471578 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.471478 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db11150-7f15-404d-a5a8-d63156df21a1" containerName="splitter-graph-999e1" Jul 06 21:09:35.471578 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.471527 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db11150-7f15-404d-a5a8-d63156df21a1" containerName="splitter-graph-999e1" Jul 06 21:09:35.474387 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.474369 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" Jul 06 21:09:35.476590 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.476566 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-d7086-serving-cert\"" Jul 06 21:09:35.476691 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.476568 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-d7086-kube-rbac-proxy-sar-config\"" Jul 06 21:09:35.484800 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.484775 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z"] Jul 06 21:09:35.525215 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.525184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f616e2-6b86-41d4-8f5d-24205f04fcef-proxy-tls\") pod \"splitter-graph-d7086-ff98df69b-m7n8z\" (UID: \"79f616e2-6b86-41d4-8f5d-24205f04fcef\") " pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" Jul 06 21:09:35.525364 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.525229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79f616e2-6b86-41d4-8f5d-24205f04fcef-openshift-service-ca-bundle\") pod \"splitter-graph-d7086-ff98df69b-m7n8z\" (UID: \"79f616e2-6b86-41d4-8f5d-24205f04fcef\") " pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" Jul 06 21:09:35.625570 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.625534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f616e2-6b86-41d4-8f5d-24205f04fcef-proxy-tls\") pod \"splitter-graph-d7086-ff98df69b-m7n8z\" (UID: \"79f616e2-6b86-41d4-8f5d-24205f04fcef\") " pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" Jul 06 21:09:35.625729 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.625580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79f616e2-6b86-41d4-8f5d-24205f04fcef-openshift-service-ca-bundle\") pod \"splitter-graph-d7086-ff98df69b-m7n8z\" (UID: \"79f616e2-6b86-41d4-8f5d-24205f04fcef\") " pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" Jul 06 21:09:35.626211 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.626191 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79f616e2-6b86-41d4-8f5d-24205f04fcef-openshift-service-ca-bundle\") pod \"splitter-graph-d7086-ff98df69b-m7n8z\" (UID: \"79f616e2-6b86-41d4-8f5d-24205f04fcef\") " pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" Jul 06 21:09:35.627887 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.627854 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f616e2-6b86-41d4-8f5d-24205f04fcef-proxy-tls\") pod \"splitter-graph-d7086-ff98df69b-m7n8z\" (UID: \"79f616e2-6b86-41d4-8f5d-24205f04fcef\") " pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" Jul 06 21:09:35.784256 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.784189 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" Jul 06 21:09:35.910588 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.910559 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z"] Jul 06 21:09:35.913647 ip-10-0-132-244 kubenswrapper[2579]: W0706 21:09:35.913618 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f616e2_6b86_41d4_8f5d_24205f04fcef.slice/crio-dc91029e61b0a2b2eec5302f58d4d402d57118b5f4c6c5cba75f346cadac5bb0 WatchSource:0}: Error finding container dc91029e61b0a2b2eec5302f58d4d402d57118b5f4c6c5cba75f346cadac5bb0: Status 404 returned error can't find the container with id dc91029e61b0a2b2eec5302f58d4d402d57118b5f4c6c5cba75f346cadac5bb0 Jul 06 21:09:35.997675 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.997641 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" event={"ID":"79f616e2-6b86-41d4-8f5d-24205f04fcef","Type":"ContainerStarted","Data":"147a3d6e4156a6635e42f5d53af86ae427b405459515c68f0d4421303a13d2b9"} Jul 06 21:09:35.997872 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.997704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" event={"ID":"79f616e2-6b86-41d4-8f5d-24205f04fcef","Type":"ContainerStarted","Data":"dc91029e61b0a2b2eec5302f58d4d402d57118b5f4c6c5cba75f346cadac5bb0"} Jul 06 21:09:35.997872 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:35.997733 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" Jul 06 21:09:36.015886 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:36.015831 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" podStartSLOduration=1.01581582 podStartE2EDuration="1.01581582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:09:36.015722603 +0000 UTC m=+2067.146085196" watchObservedRunningTime="2026-07-06 21:09:36.01581582 +0000 UTC m=+2067.146178415" Jul 06 21:09:42.006726 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:09:42.006691 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" Jul 06 21:10:09.430690 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:10:09.430577 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 21:10:09.434471 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:10:09.434453 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 21:15:09.453181 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:15:09.453070 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 21:15:09.457171 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:15:09.457152 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 21:17:50.181980 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:17:50.180004 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z"] Jul 06 21:17:50.181980 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:17:50.180311 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" podUID="79f616e2-6b86-41d4-8f5d-24205f04fcef" containerName="splitter-graph-d7086" containerID="cri-o://147a3d6e4156a6635e42f5d53af86ae427b405459515c68f0d4421303a13d2b9" gracePeriod=30 Jul 06 21:17:52.005794 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:17:52.005744 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" podUID="79f616e2-6b86-41d4-8f5d-24205f04fcef" containerName="splitter-graph-d7086" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:17:57.005698 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:17:57.005647 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" podUID="79f616e2-6b86-41d4-8f5d-24205f04fcef" containerName="splitter-graph-d7086" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:18:02.005234 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:02.005192 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" podUID="79f616e2-6b86-41d4-8f5d-24205f04fcef" containerName="splitter-graph-d7086" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:18:02.005613 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:02.005289 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" Jul 06 21:18:07.005597 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:07.005548 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" podUID="79f616e2-6b86-41d4-8f5d-24205f04fcef" containerName="splitter-graph-d7086" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:18:12.005849 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:12.005798 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" podUID="79f616e2-6b86-41d4-8f5d-24205f04fcef" containerName="splitter-graph-d7086" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:18:17.005164 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:17.005122 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" podUID="79f616e2-6b86-41d4-8f5d-24205f04fcef" containerName="splitter-graph-d7086" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:18:20.328482 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.328459 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" Jul 06 21:18:20.408135 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.408110 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f616e2-6b86-41d4-8f5d-24205f04fcef-proxy-tls\") pod \"79f616e2-6b86-41d4-8f5d-24205f04fcef\" (UID: \"79f616e2-6b86-41d4-8f5d-24205f04fcef\") " Jul 06 21:18:20.408246 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.408144 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79f616e2-6b86-41d4-8f5d-24205f04fcef-openshift-service-ca-bundle\") pod \"79f616e2-6b86-41d4-8f5d-24205f04fcef\" (UID: \"79f616e2-6b86-41d4-8f5d-24205f04fcef\") " Jul 06 21:18:20.408499 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.408470 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f616e2-6b86-41d4-8f5d-24205f04fcef-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "79f616e2-6b86-41d4-8f5d-24205f04fcef" (UID: "79f616e2-6b86-41d4-8f5d-24205f04fcef"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:18:20.409919 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.409861 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f616e2-6b86-41d4-8f5d-24205f04fcef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79f616e2-6b86-41d4-8f5d-24205f04fcef" (UID: "79f616e2-6b86-41d4-8f5d-24205f04fcef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:18:20.509373 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.509320 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f616e2-6b86-41d4-8f5d-24205f04fcef-proxy-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 21:18:20.509373 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.509343 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79f616e2-6b86-41d4-8f5d-24205f04fcef-openshift-service-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 21:18:20.891177 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.891141 2579 generic.go:356] "Generic (PLEG): container finished" podID="79f616e2-6b86-41d4-8f5d-24205f04fcef" containerID="147a3d6e4156a6635e42f5d53af86ae427b405459515c68f0d4421303a13d2b9" exitCode=0 Jul 06 21:18:20.891290 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.891178 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" event={"ID":"79f616e2-6b86-41d4-8f5d-24205f04fcef","Type":"ContainerDied","Data":"147a3d6e4156a6635e42f5d53af86ae427b405459515c68f0d4421303a13d2b9"} Jul 06 21:18:20.891290 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.891206 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" Jul 06 21:18:20.891290 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.891225 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z" event={"ID":"79f616e2-6b86-41d4-8f5d-24205f04fcef","Type":"ContainerDied","Data":"dc91029e61b0a2b2eec5302f58d4d402d57118b5f4c6c5cba75f346cadac5bb0"} Jul 06 21:18:20.891290 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.891252 2579 scope.go:117] "RemoveContainer" containerID="147a3d6e4156a6635e42f5d53af86ae427b405459515c68f0d4421303a13d2b9" Jul 06 21:18:20.900413 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.900395 2579 scope.go:117] "RemoveContainer" containerID="147a3d6e4156a6635e42f5d53af86ae427b405459515c68f0d4421303a13d2b9" Jul 06 21:18:20.900721 ip-10-0-132-244 kubenswrapper[2579]: E0706 21:18:20.900694 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147a3d6e4156a6635e42f5d53af86ae427b405459515c68f0d4421303a13d2b9\": container with ID starting with 147a3d6e4156a6635e42f5d53af86ae427b405459515c68f0d4421303a13d2b9 not found: ID does not exist" containerID="147a3d6e4156a6635e42f5d53af86ae427b405459515c68f0d4421303a13d2b9" Jul 06 21:18:20.900813 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.900729 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147a3d6e4156a6635e42f5d53af86ae427b405459515c68f0d4421303a13d2b9"} err="failed to get container status \"147a3d6e4156a6635e42f5d53af86ae427b405459515c68f0d4421303a13d2b9\": rpc error: code = NotFound desc = could not find container \"147a3d6e4156a6635e42f5d53af86ae427b405459515c68f0d4421303a13d2b9\": container with ID starting with 147a3d6e4156a6635e42f5d53af86ae427b405459515c68f0d4421303a13d2b9 not found: ID does not exist" Jul 06 21:18:20.912534 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.912509 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z"] Jul 06 21:18:20.918046 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:20.918023 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d7086-ff98df69b-m7n8z"] Jul 06 21:18:21.583517 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:18:21.583487 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79f616e2-6b86-41d4-8f5d-24205f04fcef" path="/var/lib/kubelet/pods/79f616e2-6b86-41d4-8f5d-24205f04fcef/volumes" Jul 06 21:20:09.476897 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:20:09.476802 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 21:20:09.482063 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:20:09.482041 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 21:25:09.507704 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:09.507586 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 21:25:09.513502 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:09.513478 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 21:25:14.086314 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:14.086276 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9"] Jul 06 21:25:14.086800 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:14.086521 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" podUID="0fa8cd06-104b-485e-897d-850fc8300810" containerName="switch-graph-32e32" containerID="cri-o://74ccb6930e7f90f98d865a68ec00d9dafe5968fc995eb4a6081d91572db694c0" gracePeriod=30 Jul 06 21:25:17.864257 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:17.864213 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" podUID="0fa8cd06-104b-485e-897d-850fc8300810" containerName="switch-graph-32e32" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:25:22.864594 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:22.864551 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" podUID="0fa8cd06-104b-485e-897d-850fc8300810" containerName="switch-graph-32e32" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:25:27.864596 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:27.864554 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" podUID="0fa8cd06-104b-485e-897d-850fc8300810" containerName="switch-graph-32e32" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:25:27.865132 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:27.864662 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:25:31.422705 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:31.422669 2579 ???:1] "http: TLS handshake error from 10.0.132.244:54006: EOF" Jul 06 21:25:31.428153 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:31.428122 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:32.343789 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:32.343759 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:32.863950 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:32.863915 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" podUID="0fa8cd06-104b-485e-897d-850fc8300810" containerName="switch-graph-32e32" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:25:33.247817 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:33.247742 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:34.077595 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:34.077555 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:34.914485 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:34.914447 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:35.805016 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:35.804987 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:36.659944 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:36.659915 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:37.512715 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:37.512689 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:37.864053 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:37.864017 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" podUID="0fa8cd06-104b-485e-897d-850fc8300810" containerName="switch-graph-32e32" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:25:38.409316 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:38.409280 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:39.248147 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:39.248112 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:40.089264 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:40.089242 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:40.931933 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:40.931887 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:41.775984 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:41.775938 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:42.675555 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:42.675528 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-32e32-56c9887cf-bgkh9_0fa8cd06-104b-485e-897d-850fc8300810/switch-graph-32e32/0.log" Jul 06 21:25:42.864416 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:42.864383 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" podUID="0fa8cd06-104b-485e-897d-850fc8300810" containerName="switch-graph-32e32" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:25:44.237302 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.237281 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:25:44.307212 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.307184 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa8cd06-104b-485e-897d-850fc8300810-openshift-service-ca-bundle\") pod \"0fa8cd06-104b-485e-897d-850fc8300810\" (UID: \"0fa8cd06-104b-485e-897d-850fc8300810\") " Jul 06 21:25:44.307339 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.307222 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa8cd06-104b-485e-897d-850fc8300810-proxy-tls\") pod \"0fa8cd06-104b-485e-897d-850fc8300810\" (UID: \"0fa8cd06-104b-485e-897d-850fc8300810\") " Jul 06 21:25:44.307549 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.307525 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa8cd06-104b-485e-897d-850fc8300810-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0fa8cd06-104b-485e-897d-850fc8300810" (UID: "0fa8cd06-104b-485e-897d-850fc8300810"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:25:44.309097 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.309076 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa8cd06-104b-485e-897d-850fc8300810-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0fa8cd06-104b-485e-897d-850fc8300810" (UID: "0fa8cd06-104b-485e-897d-850fc8300810"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:25:44.407877 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.407854 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa8cd06-104b-485e-897d-850fc8300810-openshift-service-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 21:25:44.407877 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.407874 2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa8cd06-104b-485e-897d-850fc8300810-proxy-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 06 21:25:44.447805 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.447772 2579 generic.go:356] "Generic (PLEG): container finished" podID="0fa8cd06-104b-485e-897d-850fc8300810" containerID="74ccb6930e7f90f98d865a68ec00d9dafe5968fc995eb4a6081d91572db694c0" exitCode=0 Jul 06 21:25:44.447898 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.447811 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" Jul 06 21:25:44.447898 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.447833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" event={"ID":"0fa8cd06-104b-485e-897d-850fc8300810","Type":"ContainerDied","Data":"74ccb6930e7f90f98d865a68ec00d9dafe5968fc995eb4a6081d91572db694c0"} Jul 06 21:25:44.447898 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.447882 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9" event={"ID":"0fa8cd06-104b-485e-897d-850fc8300810","Type":"ContainerDied","Data":"ee3c787ccd65d85d524f9f685e6e0d537dbc207884356cf0f480b673afd1eedc"} Jul 06 21:25:44.448065 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.447903 2579 scope.go:117] "RemoveContainer" containerID="74ccb6930e7f90f98d865a68ec00d9dafe5968fc995eb4a6081d91572db694c0" Jul 06 21:25:44.455617 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.455595 2579 scope.go:117] "RemoveContainer" containerID="74ccb6930e7f90f98d865a68ec00d9dafe5968fc995eb4a6081d91572db694c0" Jul 06 21:25:44.455866 ip-10-0-132-244 kubenswrapper[2579]: E0706 21:25:44.455847 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ccb6930e7f90f98d865a68ec00d9dafe5968fc995eb4a6081d91572db694c0\": container with ID starting with 74ccb6930e7f90f98d865a68ec00d9dafe5968fc995eb4a6081d91572db694c0 not found: ID does not exist" containerID="74ccb6930e7f90f98d865a68ec00d9dafe5968fc995eb4a6081d91572db694c0" Jul 06 21:25:44.455927 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.455874 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ccb6930e7f90f98d865a68ec00d9dafe5968fc995eb4a6081d91572db694c0"} err="failed to get container status \"74ccb6930e7f90f98d865a68ec00d9dafe5968fc995eb4a6081d91572db694c0\": rpc error: code = NotFound desc = could not find container \"74ccb6930e7f90f98d865a68ec00d9dafe5968fc995eb4a6081d91572db694c0\": container with ID starting with 74ccb6930e7f90f98d865a68ec00d9dafe5968fc995eb4a6081d91572db694c0 not found: ID does not exist" Jul 06 21:25:44.469277 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.469254 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9"] Jul 06 21:25:44.474385 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:44.474365 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-32e32-56c9887cf-bgkh9"] Jul 06 21:25:45.583140 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:45.583110 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fa8cd06-104b-485e-897d-850fc8300810" path="/var/lib/kubelet/pods/0fa8cd06-104b-485e-897d-850fc8300810/volumes" Jul 06 21:25:50.334322 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:50.334286 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kfgw5_2fb9f72c-98b1-4441-9c7d-57263e220a21/global-pull-secret-syncer/0.log" Jul 06 21:25:50.444235 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:50.444209 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pqs8k_8bcc5ed5-4a67-416b-bebf-93316f005012/konnectivity-agent/0.log" Jul 06 21:25:50.468203 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:50.468175 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-244.ec2.internal_9517028d6d269b9d2b51313aaffc1f4e/haproxy/0.log" Jul 06 21:25:54.231244 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:54.231219 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a250e53-7b99-424b-acaa-c50ad6d8781e/alertmanager/0.log" Jul 06 21:25:54.257423 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:54.257398 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a250e53-7b99-424b-acaa-c50ad6d8781e/config-reloader/0.log" Jul 06 21:25:54.281719 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:54.281698 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a250e53-7b99-424b-acaa-c50ad6d8781e/kube-rbac-proxy-web/0.log" Jul 06 21:25:54.303781 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:54.303756 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a250e53-7b99-424b-acaa-c50ad6d8781e/kube-rbac-proxy/0.log" Jul 06 21:25:54.326607 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:54.326587 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a250e53-7b99-424b-acaa-c50ad6d8781e/kube-rbac-proxy-metric/0.log" Jul 06 21:25:54.350424 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:54.350406 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a250e53-7b99-424b-acaa-c50ad6d8781e/prom-label-proxy/0.log" Jul 06 21:25:54.375056 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:54.375039 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a250e53-7b99-424b-acaa-c50ad6d8781e/init-config-reloader/0.log" Jul 06 21:25:54.771276 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:54.771249 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ltmvn_248602df-f8ef-438f-9a63-f9149313677b/node-exporter/0.log" Jul 06 21:25:54.800939 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:54.800912 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ltmvn_248602df-f8ef-438f-9a63-f9149313677b/kube-rbac-proxy/0.log" Jul 06 21:25:54.822565 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:54.822543 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ltmvn_248602df-f8ef-438f-9a63-f9149313677b/init-textfile/0.log" Jul 06 21:25:55.213971 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:55.213923 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54876475cf-s9rxb_5fb55fa3-ff25-4f04-8785-2c218ab4013d/telemeter-client/0.log" Jul 06 21:25:55.240189 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:55.240163 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54876475cf-s9rxb_5fb55fa3-ff25-4f04-8785-2c218ab4013d/reload/0.log" Jul 06 21:25:55.263162 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:55.263147 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54876475cf-s9rxb_5fb55fa3-ff25-4f04-8785-2c218ab4013d/kube-rbac-proxy/0.log" Jul 06 21:25:58.313474 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.313438 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh"] Jul 06 21:25:58.313838 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.313722 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa8cd06-104b-485e-897d-850fc8300810" containerName="switch-graph-32e32" Jul 06 21:25:58.313838 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.313734 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa8cd06-104b-485e-897d-850fc8300810" containerName="switch-graph-32e32" Jul 06 21:25:58.313838 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.313746 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79f616e2-6b86-41d4-8f5d-24205f04fcef" containerName="splitter-graph-d7086" Jul 06 21:25:58.313838 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.313752 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79f616e2-6b86-41d4-8f5d-24205f04fcef" containerName="splitter-graph-d7086" Jul 06 21:25:58.313838 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.313807 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79f616e2-6b86-41d4-8f5d-24205f04fcef" containerName="splitter-graph-d7086" Jul 06 21:25:58.313838 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.313815 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa8cd06-104b-485e-897d-850fc8300810" containerName="switch-graph-32e32" Jul 06 21:25:58.316799 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.316775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.318985 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.318965 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8gcgd\"/\"kube-root-ca.crt\"" Jul 06 21:25:58.319390 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.319371 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8gcgd\"/\"openshift-service-ca.crt\"" Jul 06 21:25:58.319444 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.319421 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8gcgd\"/\"default-dockercfg-vxddg\"" Jul 06 21:25:58.326881 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.326851 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh"] Jul 06 21:25:58.405318 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.405292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-proc\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.405430 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.405321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-lib-modules\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.405430 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.405343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qfm4\" (UniqueName: \"kubernetes.io/projected/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-kube-api-access-8qfm4\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.405430 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.405363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-podres\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.405564 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.405450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-sys\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.506273 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.506245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-proc\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.506397 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.506275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-lib-modules\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.506397 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.506301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qfm4\" (UniqueName: \"kubernetes.io/projected/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-kube-api-access-8qfm4\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.506397 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.506325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-podres\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.506397 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.506348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-sys\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.506397 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.506380 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-proc\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.506605 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.506441 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-sys\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.506605 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.506444 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-lib-modules\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.506605 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.506488 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-podres\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.515250 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.515223 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qfm4\" (UniqueName: \"kubernetes.io/projected/5aaa005c-961a-4fd1-baa9-a6ee02eadcbe-kube-api-access-8qfm4\") pod \"perf-node-gather-daemonset-sstkh\" (UID: \"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.627803 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.627782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:58.767510 ip-10-0-132-244 kubenswrapper[2579]: W0706 21:25:58.767470 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5aaa005c_961a_4fd1_baa9_a6ee02eadcbe.slice/crio-46f4a3adb273f5667f4108119d2acedd87f69abce15fe20fd9edf8f3a8ef5be7 WatchSource:0}: Error finding container 46f4a3adb273f5667f4108119d2acedd87f69abce15fe20fd9edf8f3a8ef5be7: Status 404 returned error can't find the container with id 46f4a3adb273f5667f4108119d2acedd87f69abce15fe20fd9edf8f3a8ef5be7 Jul 06 21:25:58.774459 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:58.774440 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh"] Jul 06 21:25:59.333512 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:59.333481 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h7rh2_18749804-aff5-4f10-a336-4562c73aff48/dns/0.log" Jul 06 21:25:59.355941 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:59.355918 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h7rh2_18749804-aff5-4f10-a336-4562c73aff48/kube-rbac-proxy/0.log" Jul 06 21:25:59.456616 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:59.456587 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nsdj2_157e3ec8-eb18-430f-b4b5-397ab0281e2a/dns-node-resolver/0.log" Jul 06 21:25:59.501821 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:59.501795 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" event={"ID":"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe","Type":"ContainerStarted","Data":"60bb9e15ade3aeb43914ed3f6aa8fe33da5adcdb8be3509c05cf6aca603b1761"} Jul 06 21:25:59.501923 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:59.501846 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" event={"ID":"5aaa005c-961a-4fd1-baa9-a6ee02eadcbe","Type":"ContainerStarted","Data":"46f4a3adb273f5667f4108119d2acedd87f69abce15fe20fd9edf8f3a8ef5be7"} Jul 06 21:25:59.501923 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:59.501879 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:25:59.520602 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:25:59.520562 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" podStartSLOduration=1.520548454 podStartE2EDuration="1.520548454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:25:59.519070338 +0000 UTC m=+3050.649432929" watchObservedRunningTime="2026-07-06 21:25:59.520548454 +0000 UTC m=+3050.650911045" Jul 06 21:26:00.051731 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:00.051713 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-whgpf_6f0d0297-cff0-4b11-b42c-af7c14b392a4/node-ca/0.log" Jul 06 21:26:00.989749 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:00.989714 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77449d57c-gl4mt_b7652700-4299-4f45-b55b-a0c4e91b4328/router/0.log" Jul 06 21:26:01.337058 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:01.336989 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7swx2_f7c05db2-dcf9-406f-a781-4ad013ae0dfc/serve-healthcheck-canary/0.log" Jul 06 21:26:01.852924 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:01.852903 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8h2x2_96e75ea1-1a37-45d1-ba33-92676762c7ce/kube-rbac-proxy/0.log" Jul 06 21:26:01.873625 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:01.873598 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8h2x2_96e75ea1-1a37-45d1-ba33-92676762c7ce/exporter/0.log" Jul 06 21:26:01.897428 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:01.897406 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8h2x2_96e75ea1-1a37-45d1-ba33-92676762c7ce/extractor/0.log" Jul 06 21:26:04.022907 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:04.022889 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-68bc99fcb8-vpqrl_6643acdf-03d1-40d6-931e-956623a065ac/manager/0.log" Jul 06 21:26:04.044318 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:04.044297 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-xpfbd_5f9c265f-2aa2-4ea5-9a3e-c36b70ff234e/manager/0.log" Jul 06 21:26:04.214300 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:04.214275 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-m4885_9bca35b4-13eb-4eef-bbf2-6d424a750967/manager/0.log" Jul 06 21:26:05.516243 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:05.516218 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-sstkh" Jul 06 21:26:10.061670 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:10.061642 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9frbs_a989d7ea-35b8-47ea-8942-a5af5d9718ab/kube-multus-additional-cni-plugins/0.log" Jul 06 21:26:10.088468 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:10.088444 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9frbs_a989d7ea-35b8-47ea-8942-a5af5d9718ab/egress-router-binary-copy/0.log" Jul 06 21:26:10.111940 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:10.111920 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9frbs_a989d7ea-35b8-47ea-8942-a5af5d9718ab/cni-plugins/0.log" Jul 06 21:26:10.135376 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:10.135351 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9frbs_a989d7ea-35b8-47ea-8942-a5af5d9718ab/bond-cni-plugin/0.log" Jul 06 21:26:10.160853 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:10.160827 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9frbs_a989d7ea-35b8-47ea-8942-a5af5d9718ab/routeoverride-cni/0.log" Jul 06 21:26:10.183406 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:10.183379 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9frbs_a989d7ea-35b8-47ea-8942-a5af5d9718ab/whereabouts-cni-bincopy/0.log" Jul 06 21:26:10.207038 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:10.207011 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9frbs_a989d7ea-35b8-47ea-8942-a5af5d9718ab/whereabouts-cni/0.log" Jul 06 21:26:10.409537 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:10.409464 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g5469_a01c55d5-4af0-4c27-8a7c-da00bae601c6/kube-multus/0.log" Jul 06 21:26:10.464721 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:10.464683 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c5bs6_47c83a08-f54e-44ff-aaf6-cd03655c8b1e/network-metrics-daemon/0.log" Jul 06 21:26:10.485920 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:10.485899 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c5bs6_47c83a08-f54e-44ff-aaf6-cd03655c8b1e/kube-rbac-proxy/0.log" Jul 06 21:26:11.765310 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:11.765287 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-controller/0.log" Jul 06 21:26:11.783865 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:11.783840 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/0.log" Jul 06 21:26:11.784070 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:11.784052 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovn-acl-logging/1.log" Jul 06 21:26:11.802757 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:11.802739 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/kube-rbac-proxy-node/0.log" Jul 06 21:26:11.822721 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:11.822704 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/kube-rbac-proxy-ovn-metrics/0.log" Jul 06 21:26:11.841009 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:11.840991 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/northd/0.log" Jul 06 21:26:11.862002 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:11.861981 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/nbdb/0.log" Jul 06 21:26:11.882397 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:11.882378 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/sbdb/0.log" Jul 06 21:26:11.940860 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:11.940843 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqkjl_366ffea7-6944-47a4-9d15-bcf3aca0756c/ovnkube-controller/0.log" Jul 06 21:26:12.975324 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:12.975297 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-m6hmf_76348c50-5f18-4580-95e2-9a92aba127dc/network-check-target-container/0.log" Jul 06 21:26:13.972265 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:13.972236 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-6wgrb_9ba8e8ab-bd92-4f3e-9266-c41227aadedb/iptables-alerter/0.log" Jul 06 21:26:14.707885 ip-10-0-132-244 kubenswrapper[2579]: I0706 21:26:14.707860 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-zwvpx_81dde24b-8630-4c06-8ce9-542aa3db5276/tuned/0.log"