Jul 16 17:31:53.156733 ip-10-0-139-103 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 17:31:53.156743 ip-10-0-139-103 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 17:31:53.156750 ip-10-0-139-103 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 17:31:53.157009 ip-10-0-139-103 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 17:32:03.168067 ip-10-0-139-103 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 16 17:32:03.168087 ip-10-0-139-103 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f3f02087e5c444b3a16636343f8e62f9 -- Jul 16 17:34:08.855938 ip-10-0-139-103 systemd[1]: Starting Kubernetes Kubelet... Jul 16 17:34:09.276238 ip-10-0-139-103 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:34:09.276238 ip-10-0-139-103 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 17:34:09.276238 ip-10-0-139-103 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:34:09.276238 ip-10-0-139-103 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 17:34:09.276238 ip-10-0-139-103 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:34:09.277130 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.277041 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 17:34:09.281560 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281535 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:34:09.281560 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281558 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:34:09.281560 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281562 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281565 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281569 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281572 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281575 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281578 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281581 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281583 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281586 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281589 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281591 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281594 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281597 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281600 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281602 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281605 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281607 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281609 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281619 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:34:09.281677 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281623 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281627 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281630 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281633 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281635 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281638 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281640 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281643 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281645 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281648 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281650 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281654 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281657 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281672 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281676 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281680 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281683 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281687 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281690 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281693 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:34:09.282135 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281696 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281699 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281701 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281704 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281707 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281709 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281713 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281717 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281719 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281722 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281725 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281728 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281731 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281734 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281737 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281739 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281742 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281744 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281747 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281749 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:34:09.282654 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281752 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281754 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281756 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281759 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281761 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281764 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281768 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281776 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281779 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281782 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281784 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281787 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281789 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281792 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281794 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281796 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281799 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281801 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281804 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:34:09.283257 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281806 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281809 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281811 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281814 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281816 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281818 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281824 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281827 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281829 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281832 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281835 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281837 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281840 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281842 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281844 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281847 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281849 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281855 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:34:09.283732 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.281858 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:34:09.284397 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284381 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:34:09.284397 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284395 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284399 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284402 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284405 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284407 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284410 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284413 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284416 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284418 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284422 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284424 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284427 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284429 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284432 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284434 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284437 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284439 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284442 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284444 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284449 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:34:09.284450 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284452 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284454 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284457 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284460 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284463 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284466 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284468 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284471 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284473 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284476 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284478 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284481 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284483 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284486 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284488 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284491 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284493 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284496 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284498 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284501 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:34:09.284962 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284504 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284507 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284509 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284512 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284514 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284517 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284519 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284521 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284524 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284527 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284529 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284531 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284534 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284536 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284539 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284542 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284545 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284547 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284549 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284552 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:34:09.285443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284554 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284557 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284559 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284562 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284564 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284567 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284571 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284574 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284577 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284580 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284582 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284586 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284588 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284592 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284594 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284597 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284600 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284604 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284606 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:34:09.285935 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284608 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284612 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284615 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284619 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284623 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284626 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284629 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284632 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284635 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284638 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284640 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284642 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284645 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284647 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284650 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284653 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284655 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284658 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.284688 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:34:09.286507 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285102 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285107 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285109 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285113 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285117 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285121 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285123 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285126 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285129 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285132 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285134 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285137 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285140 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285142 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285145 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285147 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285150 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285152 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285155 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:34:09.287251 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285158 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285161 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285164 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285166 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285168 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285171 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285173 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285176 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285178 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285180 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285183 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285186 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285189 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285192 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285194 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285197 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285199 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285202 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285205 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:34:09.287868 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285207 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285210 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285212 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285214 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285217 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285220 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285222 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285225 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285227 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285229 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285232 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285234 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285237 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285239 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285243 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285245 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285248 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285250 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285252 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285255 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:34:09.288473 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285257 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285259 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285262 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285267 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285270 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285272 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285275 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285277 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285280 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285282 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285284 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285287 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285289 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285292 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285294 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285296 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285298 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285301 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285304 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285306 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:34:09.289093 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285308 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285310 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285313 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285316 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285318 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285321 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285323 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285327 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285329 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285331 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285334 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285336 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285338 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285341 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285343 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285346 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285348 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285351 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285353 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285355 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.285358 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:34:09.289758 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285438 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285446 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285452 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285457 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285461 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285465 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285469 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285473 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285476 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285480 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285483 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285486 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285489 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285493 2582 flags.go:64] FLAG: --cgroup-root="" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285495 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285498 2582 flags.go:64] FLAG: --client-ca-file="" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285501 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285504 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285509 2582 flags.go:64] FLAG: --cluster-domain="" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285512 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285515 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285519 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285522 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285526 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 17:34:09.290316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285529 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285532 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285535 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285538 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285542 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285545 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285548 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285551 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285555 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285558 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285561 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285564 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285567 2582 flags.go:64] FLAG: --enable-server="true" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285570 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285574 2582 flags.go:64] FLAG: --event-burst="100" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285578 2582 flags.go:64] FLAG: --event-qps="50" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285580 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285583 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285586 2582 flags.go:64] FLAG: --eviction-hard="" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285590 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285593 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285596 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285599 2582 flags.go:64] FLAG: --eviction-soft="" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285601 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285608 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 17:34:09.291001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285611 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285614 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285617 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285620 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285623 2582 flags.go:64] FLAG: --feature-gates="" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285627 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285630 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285633 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285636 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285639 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285642 2582 flags.go:64] FLAG: --help="false" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285645 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285648 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285651 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285653 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285657 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285674 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285677 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285680 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285683 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285686 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285689 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285694 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285697 2582 flags.go:64] FLAG: --kube-reserved="" Jul 16 17:34:09.291616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285700 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285702 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285705 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285708 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285711 2582 flags.go:64] FLAG: --lock-file="" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285714 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285717 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285721 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285727 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285730 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285732 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285735 2582 flags.go:64] FLAG: --logging-format="text" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285738 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285742 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285745 2582 flags.go:64] FLAG: --manifest-url="" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285748 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285752 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285755 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285760 2582 flags.go:64] FLAG: --max-pods="110" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285763 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285766 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285768 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285771 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285774 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285777 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 17:34:09.292251 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285780 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285787 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285791 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285793 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285796 2582 flags.go:64] FLAG: --pod-cidr="" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285800 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285807 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285810 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285813 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285816 2582 flags.go:64] FLAG: --port="10250" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285819 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285822 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06cb033f0b9bc1238" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285825 2582 flags.go:64] FLAG: --qos-reserved="" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285828 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285831 2582 flags.go:64] FLAG: --register-node="true" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285835 2582 flags.go:64] FLAG: --register-with-taints="" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285838 2582 flags.go:64] FLAG: --registry-burst="10" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285841 2582 flags.go:64] FLAG: --registry-qps="5" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285844 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285847 2582 flags.go:64] FLAG: --reserved-memory="" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285851 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285854 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285857 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285860 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285863 2582 flags.go:64] FLAG: --runonce="false" Jul 16 17:34:09.292912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285866 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285869 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285871 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285874 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285877 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285880 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285883 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285886 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285889 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285892 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285895 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285898 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285902 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285905 2582 flags.go:64] FLAG: --system-cgroups="" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285907 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285913 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285916 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285918 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285923 2582 flags.go:64] FLAG: --tls-min-version="" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285925 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285928 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285931 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285935 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285938 2582 flags.go:64] FLAG: --v="2" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285942 2582 flags.go:64] FLAG: --version="false" Jul 16 17:34:09.293516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285947 2582 flags.go:64] FLAG: --vmodule="" Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285950 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.285954 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286845 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286860 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286865 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286870 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286875 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286879 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286884 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286887 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286892 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286901 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286905 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286909 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286913 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286918 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286922 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286926 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286933 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:34:09.294197 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286937 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286941 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286946 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286951 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286955 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286964 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286968 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286973 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286977 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286983 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286987 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286992 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.286996 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287001 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287005 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287010 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287017 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287028 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287033 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287037 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:34:09.294756 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287041 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287046 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287051 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287055 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287059 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287064 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287068 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287073 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287077 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287082 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287091 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287100 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287105 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287109 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287113 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287117 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287124 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287130 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287134 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:34:09.295254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287139 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287143 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287170 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287196 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287203 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287209 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287214 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287222 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287264 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287373 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287380 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287385 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287390 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287395 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287400 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287405 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287409 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287415 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287419 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:34:09.295736 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287424 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287428 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287433 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287437 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287442 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287446 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287451 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287455 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287459 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287463 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287467 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287472 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287476 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287480 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287484 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287488 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287496 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287503 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287507 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287511 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:34:09.296206 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287516 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287521 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287526 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.287531 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.288070 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.295851 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.295869 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.296639 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.296648 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296733 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296739 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296743 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296746 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296749 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296751 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:34:09.296804 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296755 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296757 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296760 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296762 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296765 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296768 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296770 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296774 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296778 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296781 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296784 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296787 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296789 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296792 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296795 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296798 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296801 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296804 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296806 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296809 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:34:09.297184 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296812 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296814 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296817 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296820 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296822 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296824 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296827 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296830 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296832 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296834 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296837 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296839 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296842 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296844 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296848 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296851 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296854 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296856 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296859 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296862 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:34:09.297685 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296864 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296867 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296869 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296872 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296874 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296876 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296879 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296881 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296885 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296888 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296890 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296893 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296895 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296897 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296901 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296905 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296908 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296910 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:34:09.298173 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296914 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296917 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296920 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296923 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296926 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296928 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296931 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296934 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296937 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296940 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296942 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296945 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296948 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296950 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296953 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296955 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296958 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296960 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296963 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296965 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:34:09.298616 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296968 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296970 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296972 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296976 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296978 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296981 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296984 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296986 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296989 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296991 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296994 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296997 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.296999 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297002 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297004 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:34:09.299112 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.297009 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297133 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297139 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297142 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297145 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297148 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297151 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297153 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297156 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297158 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297161 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297163 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297166 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297168 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297171 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297173 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297175 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297178 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297180 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297183 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:34:09.299479 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297185 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297188 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297192 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297194 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297197 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297199 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297202 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297205 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297207 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297210 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297212 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297235 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297238 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297241 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297244 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297247 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297250 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297254 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297257 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297260 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:34:09.300010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297263 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297266 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297268 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297271 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297274 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297276 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297279 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297281 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297284 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297286 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297289 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297291 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297294 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297296 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297299 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297302 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297306 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297309 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297313 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:34:09.300511 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297315 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297318 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297321 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297324 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297326 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297328 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297332 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297334 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297337 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297339 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297342 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297344 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297347 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297349 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297352 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297354 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297357 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297359 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297362 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297364 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:34:09.300996 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297367 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297369 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297371 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297375 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297379 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297381 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297384 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297386 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297389 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297392 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297395 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297398 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297400 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297403 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297406 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297408 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297412 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297416 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297419 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:34:09.301530 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297421 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:34:09.302003 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:09.297424 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:34:09.302003 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.297428 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:34:09.302003 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.297561 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 17:34:09.302003 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.299843 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 17:34:09.302003 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.301145 2582 server.go:1019] "Starting client certificate rotation" Jul 16 17:34:09.302003 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.301253 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 17:34:09.302003 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.301300 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 17:34:09.303451 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.303434 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 17:34:09.304682 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.304652 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 17:34:09.317897 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.317865 2582 log.go:25] "Validated CRI v1 runtime API" Jul 16 17:34:09.321173 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.321147 2582 log.go:25] "Validated CRI v1 image API" Jul 16 17:34:09.322104 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.322086 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 17:34:09.325884 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.325862 2582 fs.go:135] Filesystem UUIDs: map[0b776f63-8e82-4584-bce1-e71754c91c35:/dev/nvme0n1p3 404760c3-7945-4909-b867-973716c3678b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 16 17:34:09.325945 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.325884 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 17:34:09.329379 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.329351 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 17:34:09.331170 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.331043 2582 manager.go:216] Machine: {Timestamp:2026-07-16 17:34:09.329688506 +0000 UTC m=+0.351725979 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091555 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea557f9981aab9ad3d0011b6e96ca SystemUUID:ec2ea557-f998-1aab-9ad3-d0011b6e96ca BootID:f3f02087-e5c4-44b3-a166-36343f8e62f9 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:59:8d:ff:a5:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:59:8d:ff:a5:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:a7:a2:78:bd:1f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 17:34:09.331170 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.331164 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 17:34:09.331294 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.331262 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 17:34:09.336605 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.336567 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 17:34:09.336778 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.336609 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-103.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 17:34:09.337393 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.337382 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 17:34:09.337438 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.337398 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 17:34:09.337438 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.337406 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 17:34:09.338092 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.338080 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 17:34:09.338929 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.338916 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 17:34:09.341691 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.341657 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 16 17:34:09.341841 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.341831 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 17:34:09.341873 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.341864 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 16 17:34:09.341929 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.341875 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 17:34:09.341929 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.341890 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 17:34:09.341929 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.341905 2582 kubelet.go:398] "Adding apiserver pod source" Jul 16 17:34:09.341929 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.341915 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 17:34:09.342402 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.342389 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 17:34:09.342435 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.342411 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 17:34:09.345871 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.345853 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 17:34:09.347159 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.347144 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 17:34:09.347210 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.347176 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 17:34:09.349951 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.349934 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 17:34:09.349951 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.349951 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 17:34:09.350049 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.349958 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 17:34:09.350049 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.349964 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 17:34:09.350049 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.349972 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 17:34:09.350049 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.349977 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 17:34:09.350049 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.349985 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 17:34:09.350049 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.349990 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 17:34:09.350049 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.349999 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 17:34:09.350049 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.350004 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 17:34:09.350049 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.350018 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 17:34:09.350049 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.350024 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 17:34:09.351161 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.351132 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 17:34:09.351198 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.351172 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 17:34:09.352553 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.352539 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 17:34:09.352599 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.352555 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 17:34:09.354300 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.354277 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-103.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 17:34:09.355965 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.355951 2582 server.go:1268] "Started kubelet" Jul 16 17:34:09.356092 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.356061 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 17:34:09.356156 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.356106 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 17:34:09.356200 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.356181 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 17:34:09.357158 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.357139 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 17:34:09.357533 ip-10-0-139-103 systemd[1]: Started Kubernetes Kubelet. Jul 16 17:34:09.358893 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.358880 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 16 17:34:09.362641 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.362615 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 17:34:09.362815 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.362800 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 17:34:09.362874 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.362843 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 17:34:09.362933 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.362912 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-103.ec2.internal\" not found" Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.363445 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.363447 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.363473 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.363571 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.363581 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.363592 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.363606 2582 factory.go:55] Registering systemd factory Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.363616 2582 factory.go:223] Registration of the systemd container factory successfully Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.363826 2582 factory.go:153] Registering CRI-O factory Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.363837 2582 factory.go:223] Registration of the crio container factory successfully Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.363857 2582 factory.go:103] Registering Raw factory Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.363871 2582 manager.go:1196] Started watching for new ooms in manager Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.364248 2582 manager.go:349] Starting recovery of all containers Jul 16 17:34:09.365006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.364312 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6wks5" Jul 16 17:34:09.371259 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.371021 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 17:34:09.371259 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.371113 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 17:34:09.372225 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.371311 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-103.ec2.internal.18c2d59791da307a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-103.ec2.internal,UID:ip-10-0-139-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-103.ec2.internal,},FirstTimestamp:2026-07-16 17:34:09.35591129 +0000 UTC m=+0.377948742,LastTimestamp:2026-07-16 17:34:09.35591129 +0000 UTC m=+0.377948742,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-103.ec2.internal,}" Jul 16 17:34:09.372962 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.372932 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6wks5" Jul 16 17:34:09.374089 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.374069 2582 manager.go:354] Recovery completed Jul 16 17:34:09.379083 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.379066 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:34:09.382441 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.382420 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:34:09.382536 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.382459 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:34:09.382536 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.382473 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:34:09.383194 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.383178 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 17:34:09.383194 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.383192 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 17:34:09.383340 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.383195 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:34:09.383340 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.383207 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 16 17:34:09.384102 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.384085 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:34:09.384182 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.384114 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:34:09.384182 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.384144 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:34:09.384332 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.384319 2582 policy_none.go:49] "None policy: Start" Jul 16 17:34:09.384381 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.384341 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 17:34:09.384381 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.384353 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 17:34:09.385359 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.385344 2582 policy_none.go:47] "Start" Jul 16 17:34:09.420383 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.420351 2582 manager.go:337] "Starting Device Plugin manager" Jul 16 17:34:09.420518 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.420402 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 17:34:09.420518 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.420415 2582 server.go:87] "Starting device plugin registration server" Jul 16 17:34:09.420755 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.420739 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 17:34:09.420808 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.420756 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 17:34:09.420890 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.420876 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 17:34:09.420989 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.420971 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 17:34:09.420989 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.420985 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 17:34:09.421521 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.421501 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 17:34:09.421616 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.421566 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-103.ec2.internal\" not found" Jul 16 17:34:09.521166 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.521132 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:34:09.524764 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.524748 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:34:09.524855 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.524779 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:34:09.524855 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.524790 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:34:09.524855 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.524817 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.533472 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.533404 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.533472 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.533431 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-103.ec2.internal\": node \"ip-10-0-139-103.ec2.internal\" not found" Jul 16 17:34:09.550855 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.550822 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-103.ec2.internal\" not found" Jul 16 17:34:09.573021 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.572990 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 17:34:09.574317 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.574295 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 17:34:09.574419 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.574337 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 17:34:09.574419 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.574360 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 17:34:09.574419 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.574401 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 17:34:09.576597 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.576572 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:34:09.651335 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.651298 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-103.ec2.internal\" not found" Jul 16 17:34:09.674713 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.674689 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-103.ec2.internal"] Jul 16 17:34:09.674821 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.674780 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:34:09.676490 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.676469 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:34:09.676598 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.676501 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:34:09.676598 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.676511 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:34:09.677840 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.677828 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:34:09.677951 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.677937 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.678037 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.677975 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:34:09.678610 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.678595 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:34:09.678691 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.678624 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:34:09.678691 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.678634 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:34:09.678691 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.678595 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:34:09.678791 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.678703 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:34:09.678791 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.678717 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:34:09.679805 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.679790 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.679877 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.679814 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:34:09.680556 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.680541 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:34:09.680622 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.680567 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:34:09.680622 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.680578 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:34:09.713655 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.713627 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-103.ec2.internal\" not found" node="ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.718309 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.718292 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-103.ec2.internal\" not found" node="ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.752132 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.752104 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-103.ec2.internal\" not found" Jul 16 17:34:09.764708 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.764679 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/510a81c26cef2cf6d90419e4745d7adc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal\" (UID: \"510a81c26cef2cf6d90419e4745d7adc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.764794 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.764718 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/510a81c26cef2cf6d90419e4745d7adc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal\" (UID: \"510a81c26cef2cf6d90419e4745d7adc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.764794 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.764735 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8cc93a3e3b0301bf02b2011bdc86d70-config\") pod \"kube-apiserver-proxy-ip-10-0-139-103.ec2.internal\" (UID: \"b8cc93a3e3b0301bf02b2011bdc86d70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.852698 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.852610 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-103.ec2.internal\" not found" Jul 16 17:34:09.864894 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.864866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/510a81c26cef2cf6d90419e4745d7adc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal\" (UID: \"510a81c26cef2cf6d90419e4745d7adc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.864969 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.864901 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/510a81c26cef2cf6d90419e4745d7adc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal\" (UID: \"510a81c26cef2cf6d90419e4745d7adc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.864969 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.864918 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8cc93a3e3b0301bf02b2011bdc86d70-config\") pod \"kube-apiserver-proxy-ip-10-0-139-103.ec2.internal\" (UID: \"b8cc93a3e3b0301bf02b2011bdc86d70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.864969 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.864941 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/510a81c26cef2cf6d90419e4745d7adc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal\" (UID: \"510a81c26cef2cf6d90419e4745d7adc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.864969 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.864955 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8cc93a3e3b0301bf02b2011bdc86d70-config\") pod \"kube-apiserver-proxy-ip-10-0-139-103.ec2.internal\" (UID: \"b8cc93a3e3b0301bf02b2011bdc86d70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.864969 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:09.864957 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/510a81c26cef2cf6d90419e4745d7adc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal\" (UID: \"510a81c26cef2cf6d90419e4745d7adc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal" Jul 16 17:34:09.953091 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:09.953058 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-103.ec2.internal\" not found" Jul 16 17:34:11.021385 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.021342 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal" Jul 16 17:34:11.025553 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.025223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.ec2.internal" Jul 16 17:34:11.059022 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:11.058979 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-103.ec2.internal\" not found" Jul 16 17:34:11.159624 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:11.159534 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-103.ec2.internal\" not found" Jul 16 17:34:11.260084 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:11.260040 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-103.ec2.internal\" not found" Jul 16 17:34:11.306607 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.306582 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 17:34:11.307242 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.306767 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 17:34:11.360226 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:11.360190 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-103.ec2.internal\" not found" Jul 16 17:34:11.381865 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.381831 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 17:29:10 +0000 UTC" deadline="2027-12-16 23:29:43.072049134 +0000 UTC" Jul 16 17:34:11.381865 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.381861 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12437h55m31.690191601s" Jul 16 17:34:11.460705 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:11.460623 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-103.ec2.internal\" not found" Jul 16 17:34:11.535924 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.535899 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:34:11.568007 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.567976 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal" Jul 16 17:34:11.583420 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.583391 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 17:34:11.584897 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.584880 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.ec2.internal" Jul 16 17:34:11.593596 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.593569 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 17:34:11.637308 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:11.637274 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cc93a3e3b0301bf02b2011bdc86d70.slice/crio-7c30513d0c4cfbbbf9d0cd8e951e1bb1a3e36cb92decba58533a3e38d77bbac5 WatchSource:0}: Error finding container 7c30513d0c4cfbbbf9d0cd8e951e1bb1a3e36cb92decba58533a3e38d77bbac5: Status 404 returned error can't find the container with id 7c30513d0c4cfbbbf9d0cd8e951e1bb1a3e36cb92decba58533a3e38d77bbac5 Jul 16 17:34:11.642246 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.642224 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:34:11.739625 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.739541 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:34:11.858278 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:11.858255 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:34:12.298634 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.298605 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:34:12.347986 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.347959 2582 apiserver.go:52] "Watching apiserver" Jul 16 17:34:12.359825 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.359788 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 17:34:12.360262 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.360235 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hfrlv","kube-system/konnectivity-agent-7kc7q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal","openshift-multus/multus-qj6ls","openshift-multus/network-metrics-daemon-5qrcw","kube-system/kube-apiserver-proxy-ip-10-0-139-103.ec2.internal","openshift-cluster-node-tuning-operator/tuned-h6xsw","openshift-dns/node-resolver-d7b2n","openshift-image-registry/node-ca-4ptfz","openshift-multus/multus-additional-cni-plugins-ct6j5","openshift-network-diagnostics/network-check-target-x2vn2","openshift-network-operator/iptables-alerter-855sl","openshift-ovn-kubernetes/ovnkube-node-4lfqt"] Jul 16 17:34:12.362184 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.361766 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-855sl" Jul 16 17:34:12.362985 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.362964 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7kc7q" Jul 16 17:34:12.364274 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.364224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.364498 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.364461 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:34:12.364706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.364684 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 17:34:12.364844 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.364705 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 17:34:12.364844 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.364729 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vj8sn\"" Jul 16 17:34:12.365673 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.365635 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 17:34:12.365902 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.365886 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4vpbd\"" Jul 16 17:34:12.366224 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.366087 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 17:34:12.366578 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.366560 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 17:34:12.366930 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.366756 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 17:34:12.366930 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.366801 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 17:34:12.366930 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.366848 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l84nq\"" Jul 16 17:34:12.368069 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.368049 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:12.368167 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.368123 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:12.368338 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.368317 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 17:34:12.369505 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.369486 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.369935 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.369919 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:12.370035 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.370010 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:12.370922 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.370904 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.372161 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.372140 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9vd4k\"" Jul 16 17:34:12.372387 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.372370 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 17:34:12.373197 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.373179 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 17:34:12.373454 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.373438 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 17:34:12.373653 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.373638 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 17:34:12.373857 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.373849 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 17:34:12.374022 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.374007 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 17:34:12.374496 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.374368 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h525p\"" Jul 16 17:34:12.374572 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.374511 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 17:34:12.374636 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.374569 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 17:34:12.374757 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.374740 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.374835 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.374818 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 17:34:12.375296 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.375277 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 17:34:12.375493 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.375459 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d7b2n" Jul 16 17:34:12.375892 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.375868 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-dbus\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:12.375991 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.375903 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4e92fef-28b7-4d9e-af27-d22d54ca66a6-host-slash\") pod \"iptables-alerter-855sl\" (UID: \"e4e92fef-28b7-4d9e-af27-d22d54ca66a6\") " pod="openshift-network-operator/iptables-alerter-855sl" Jul 16 17:34:12.375991 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.375927 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-multus-conf-dir\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.375991 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.375951 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-run-multus-certs\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.375991 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.375978 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-socket-dir\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.376196 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376003 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:12.376196 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376027 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-run-k8s-cni-cncf-io\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.376196 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376050 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-sys-fs\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.376196 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376092 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twrb4\" (UniqueName: \"kubernetes.io/projected/728b9423-5d3a-471d-8eb6-f23a19614cd0-kube-api-access-twrb4\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.376196 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376115 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-multus-cni-dir\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.376196 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376137 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-var-lib-cni-bin\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.376196 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376161 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-etc-kubernetes\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.376196 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376187 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-kubelet-config\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376211 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-run-netns\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4ptfz" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376246 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-var-lib-kubelet\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376272 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2cd9\" (UniqueName: \"kubernetes.io/projected/dc349e33-d51c-40bf-8890-392c3393299d-kube-api-access-k2cd9\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376295 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-system-cni-dir\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376321 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-cnibin\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376343 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376384 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-device-dir\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-os-release\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376452 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-cni-binary-copy\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376496 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljc5r\" (UniqueName: \"kubernetes.io/projected/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-kube-api-access-ljc5r\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376533 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30219802-706a-47e2-9bf1-8942aade426c-agent-certs\") pod \"konnectivity-agent-7kc7q\" (UID: \"30219802-706a-47e2-9bf1-8942aade426c\") " pod="kube-system/konnectivity-agent-7kc7q" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376559 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30219802-706a-47e2-9bf1-8942aade426c-konnectivity-ca\") pod \"konnectivity-agent-7kc7q\" (UID: \"30219802-706a-47e2-9bf1-8942aade426c\") " pod="kube-system/konnectivity-agent-7kc7q" Jul 16 17:34:12.376586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376583 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-registration-dir\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.377224 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376605 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-etc-selinux\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.377224 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376647 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-multus-socket-dir-parent\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.377224 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376671 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-var-lib-cni-multus\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.377224 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376698 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-hostroot\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.377224 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376739 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:12.377224 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376763 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e4e92fef-28b7-4d9e-af27-d22d54ca66a6-iptables-alerter-script\") pod \"iptables-alerter-855sl\" (UID: \"e4e92fef-28b7-4d9e-af27-d22d54ca66a6\") " pod="openshift-network-operator/iptables-alerter-855sl" Jul 16 17:34:12.377224 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-multus-daemon-config\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.377224 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.376820 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znvnb\" (UniqueName: \"kubernetes.io/projected/e4e92fef-28b7-4d9e-af27-d22d54ca66a6-kube-api-access-znvnb\") pod \"iptables-alerter-855sl\" (UID: \"e4e92fef-28b7-4d9e-af27-d22d54ca66a6\") " pod="openshift-network-operator/iptables-alerter-855sl" Jul 16 17:34:12.377224 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.377151 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:34:12.377707 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.377416 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cnvkw\"" Jul 16 17:34:12.377707 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.377495 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 17:34:12.377939 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.377843 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wfgd5\"" Jul 16 17:34:12.377939 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.377845 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 17:34:12.379424 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.378771 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r6mcv\"" Jul 16 17:34:12.379424 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.378797 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 17:34:12.379424 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.378978 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 17:34:12.379691 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.379673 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.380884 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.379996 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 17:34:12.380884 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.380338 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 17:34:12.381134 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.381114 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:12.381217 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.381196 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:12.383651 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.383639 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 17:34:12.383832 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.383813 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 17:34:12.384116 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.384052 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mjnrc\"" Jul 16 17:34:12.388049 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.388024 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 17:34:12.417146 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.417122 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j7m4f" Jul 16 17:34:12.434208 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.434185 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j7m4f" Jul 16 17:34:12.470316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.470276 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 17:34:12.477039 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.476976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-run-netns\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.477039 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477017 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-var-lib-kubelet\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.477039 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477044 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2cd9\" (UniqueName: \"kubernetes.io/projected/dc349e33-d51c-40bf-8890-392c3393299d-kube-api-access-k2cd9\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:12.477319 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-etc-openvswitch\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.477319 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-cni-bin\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.477319 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477131 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-kubernetes\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.477319 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477164 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-system-cni-dir\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.477319 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477186 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-cnibin\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.477319 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477209 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-systemd-units\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.477319 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477235 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-run-netns\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.477319 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477259 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-log-socket\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.477319 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477292 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-sysctl-conf\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.477319 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477293 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-run-netns\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477310 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477351 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-os-release\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477353 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-var-lib-kubelet\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477369 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-cni-binary-copy\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477385 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30219802-706a-47e2-9bf1-8942aade426c-agent-certs\") pod \"konnectivity-agent-7kc7q\" (UID: \"30219802-706a-47e2-9bf1-8942aade426c\") " pod="kube-system/konnectivity-agent-7kc7q" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477401 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-node-log\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-sysctl-d\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477434 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-multus-socket-dir-parent\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477457 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-var-lib-cni-multus\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477487 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-hostroot\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477513 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e4e92fef-28b7-4d9e-af27-d22d54ca66a6-iptables-alerter-script\") pod \"iptables-alerter-855sl\" (UID: \"e4e92fef-28b7-4d9e-af27-d22d54ca66a6\") " pod="openshift-network-operator/iptables-alerter-855sl" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477554 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-sysconfig\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-run\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477602 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-host\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477635 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvhk\" (UniqueName: \"kubernetes.io/projected/52d2c0de-2ebc-45eb-ad30-aad441699edd-kube-api-access-2jvhk\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477665 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-lib-modules\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.477731 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477689 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782eb993-d0db-4c25-b990-34dbe4d7a5b9-tmp-dir\") pod \"node-resolver-d7b2n\" (UID: \"782eb993-d0db-4c25-b990-34dbe4d7a5b9\") " pod="openshift-dns/node-resolver-d7b2n" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbf4d\" (UniqueName: \"kubernetes.io/projected/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-kube-api-access-rbf4d\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477743 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-dbus\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477768 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4e92fef-28b7-4d9e-af27-d22d54ca66a6-host-slash\") pod \"iptables-alerter-855sl\" (UID: \"e4e92fef-28b7-4d9e-af27-d22d54ca66a6\") " pod="openshift-network-operator/iptables-alerter-855sl" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477793 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5fa87729-4cd3-4453-a444-9c59a6514753-serviceca\") pod \"node-ca-4ptfz\" (UID: \"5fa87729-4cd3-4453-a444-9c59a6514753\") " pod="openshift-image-registry/node-ca-4ptfz" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-multus-conf-dir\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477857 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-run-k8s-cni-cncf-io\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477896 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-kubelet\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477911 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-tuned\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477931 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.477961 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478020 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/807cdaf2-aa32-4788-9661-fb1b56a34fc2-ovnkube-script-lib\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478005 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-hostroot\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478074 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-sys-fs\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrb4\" (UniqueName: \"kubernetes.io/projected/728b9423-5d3a-471d-8eb6-f23a19614cd0-kube-api-access-twrb4\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478135 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-system-cni-dir\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.478486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478147 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-var-lib-cni-bin\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478173 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-etc-kubernetes\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478195 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-cnibin\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478198 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m22dw\" (UniqueName: \"kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw\") pod \"network-check-target-x2vn2\" (UID: \"458ea307-1c97-4cda-8723-97e047f76a50\") " pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478237 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-systemd\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478269 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-cni-binary-copy\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478305 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-kubelet-config\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478331 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478356 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-cni-netd\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478410 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ggvm\" (UniqueName: \"kubernetes.io/projected/807cdaf2-aa32-4788-9661-fb1b56a34fc2-kube-api-access-9ggvm\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478436 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/807cdaf2-aa32-4788-9661-fb1b56a34fc2-env-overrides\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478464 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-device-dir\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478511 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljc5r\" (UniqueName: \"kubernetes.io/projected/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-kube-api-access-ljc5r\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478539 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30219802-706a-47e2-9bf1-8942aade426c-konnectivity-ca\") pod \"konnectivity-agent-7kc7q\" (UID: \"30219802-706a-47e2-9bf1-8942aade426c\") " pod="kube-system/konnectivity-agent-7kc7q" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478564 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-slash\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478589 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-system-cni-dir\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.479051 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478615 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-os-release\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478655 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-registration-dir\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478682 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-etc-selinux\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478734 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/807cdaf2-aa32-4788-9661-fb1b56a34fc2-ovn-node-metrics-cert\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478760 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5fa87729-4cd3-4453-a444-9c59a6514753-host\") pod \"node-ca-4ptfz\" (UID: \"5fa87729-4cd3-4453-a444-9c59a6514753\") " pod="openshift-image-registry/node-ca-4ptfz" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478785 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-multus-daemon-config\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znvnb\" (UniqueName: \"kubernetes.io/projected/e4e92fef-28b7-4d9e-af27-d22d54ca66a6-kube-api-access-znvnb\") pod \"iptables-alerter-855sl\" (UID: \"e4e92fef-28b7-4d9e-af27-d22d54ca66a6\") " pod="openshift-network-operator/iptables-alerter-855sl" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478828 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e4e92fef-28b7-4d9e-af27-d22d54ca66a6-iptables-alerter-script\") pod \"iptables-alerter-855sl\" (UID: \"e4e92fef-28b7-4d9e-af27-d22d54ca66a6\") " pod="openshift-network-operator/iptables-alerter-855sl" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478836 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-run-systemd\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478866 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-var-lib-openvswitch\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478895 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-run-ovn\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-sys\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478951 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgcft\" (UniqueName: \"kubernetes.io/projected/782eb993-d0db-4c25-b990-34dbe4d7a5b9-kube-api-access-cgcft\") pod \"node-resolver-d7b2n\" (UID: \"782eb993-d0db-4c25-b990-34dbe4d7a5b9\") " pod="openshift-dns/node-resolver-d7b2n" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.478977 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/782eb993-d0db-4c25-b990-34dbe4d7a5b9-hosts-file\") pod \"node-resolver-d7b2n\" (UID: \"782eb993-d0db-4c25-b990-34dbe4d7a5b9\") " pod="openshift-dns/node-resolver-d7b2n" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-run-multus-certs\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-socket-dir\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.479665 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479060 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-run-openvswitch\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479087 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-var-lib-kubelet\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479116 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-multus-cni-dir\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479121 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-multus-conf-dir\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-run-ovn-kubernetes\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479202 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479219 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479229 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/807cdaf2-aa32-4788-9661-fb1b56a34fc2-ovnkube-config\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479255 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnd65\" (UniqueName: \"kubernetes.io/projected/5fa87729-4cd3-4453-a444-9c59a6514753-kube-api-access-nnd65\") pod \"node-ca-4ptfz\" (UID: \"5fa87729-4cd3-4453-a444-9c59a6514753\") " pod="openshift-image-registry/node-ca-4ptfz" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479279 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-modprobe-d\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479284 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-os-release\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479304 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52d2c0de-2ebc-45eb-ad30-aad441699edd-tmp\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479325 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-cnibin\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.479452 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.479579 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret podName:503d02a0-3e8a-4786-bf7e-c4c0bec8dbda nodeName:}" failed. No retries permitted until 2026-07-16 17:34:12.979550465 +0000 UTC m=+2.996779788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret") pod "global-pull-secret-syncer-hfrlv" (UID: "503d02a0-3e8a-4786-bf7e-c4c0bec8dbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479631 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-run-k8s-cni-cncf-io\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479756 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-sys-fs\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.480218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479811 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-cni-binary-copy\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.480977 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479967 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-var-lib-cni-bin\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.480977 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.480007 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-etc-kubernetes\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.480977 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.480063 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-multus-cni-dir\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.480977 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.480090 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 17:34:12.480977 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.480539 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-registration-dir\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.480977 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.480644 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-multus-socket-dir-parent\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.480977 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.480696 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-var-lib-cni-multus\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.480977 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.480735 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-kubelet-config\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:12.480977 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.480807 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-device-dir\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.482181 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.481407 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30219802-706a-47e2-9bf1-8942aade426c-konnectivity-ca\") pod \"konnectivity-agent-7kc7q\" (UID: \"30219802-706a-47e2-9bf1-8942aade426c\") " pod="kube-system/konnectivity-agent-7kc7q" Jul 16 17:34:12.482181 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.481524 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-etc-selinux\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.482181 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.481539 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-host-run-multus-certs\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.482181 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.481528 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/728b9423-5d3a-471d-8eb6-f23a19614cd0-socket-dir\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.482181 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.481603 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:12.482181 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.479060 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-dbus\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:12.482181 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.481661 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4e92fef-28b7-4d9e-af27-d22d54ca66a6-host-slash\") pod \"iptables-alerter-855sl\" (UID: \"e4e92fef-28b7-4d9e-af27-d22d54ca66a6\") " pod="openshift-network-operator/iptables-alerter-855sl" Jul 16 17:34:12.482181 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.481749 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs podName:dc349e33-d51c-40bf-8890-392c3393299d nodeName:}" failed. No retries permitted until 2026-07-16 17:34:12.981729245 +0000 UTC m=+2.998958566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs") pod "network-metrics-daemon-5qrcw" (UID: "dc349e33-d51c-40bf-8890-392c3393299d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:12.482181 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.481997 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-multus-daemon-config\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.484059 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.484034 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30219802-706a-47e2-9bf1-8942aade426c-agent-certs\") pod \"konnectivity-agent-7kc7q\" (UID: \"30219802-706a-47e2-9bf1-8942aade426c\") " pod="kube-system/konnectivity-agent-7kc7q" Jul 16 17:34:12.487721 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.487689 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2cd9\" (UniqueName: \"kubernetes.io/projected/dc349e33-d51c-40bf-8890-392c3393299d-kube-api-access-k2cd9\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:12.490907 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.490660 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljc5r\" (UniqueName: \"kubernetes.io/projected/e59c2366-fc5e-4e31-a94f-5a39c99dc3d5-kube-api-access-ljc5r\") pod \"multus-qj6ls\" (UID: \"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5\") " pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.490907 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.490861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twrb4\" (UniqueName: \"kubernetes.io/projected/728b9423-5d3a-471d-8eb6-f23a19614cd0-kube-api-access-twrb4\") pod \"aws-ebs-csi-driver-node-ql6hb\" (UID: \"728b9423-5d3a-471d-8eb6-f23a19614cd0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.491878 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.491853 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znvnb\" (UniqueName: \"kubernetes.io/projected/e4e92fef-28b7-4d9e-af27-d22d54ca66a6-kube-api-access-znvnb\") pod \"iptables-alerter-855sl\" (UID: \"e4e92fef-28b7-4d9e-af27-d22d54ca66a6\") " pod="openshift-network-operator/iptables-alerter-855sl" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.579762 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-modprobe-d\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.579818 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52d2c0de-2ebc-45eb-ad30-aad441699edd-tmp\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.579861 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-cnibin\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.579905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-etc-openvswitch\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.579936 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-cni-bin\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.579964 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-kubernetes\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580008 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-systemd-units\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580036 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-run-netns\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580074 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-log-socket\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-sysctl-conf\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580138 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-node-log\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-sysctl-d\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-sysconfig\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580251 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-run\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580277 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-host\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580301 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvhk\" (UniqueName: \"kubernetes.io/projected/52d2c0de-2ebc-45eb-ad30-aad441699edd-kube-api-access-2jvhk\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580357 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-lib-modules\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.580693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580393 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782eb993-d0db-4c25-b990-34dbe4d7a5b9-tmp-dir\") pod \"node-resolver-d7b2n\" (UID: \"782eb993-d0db-4c25-b990-34dbe4d7a5b9\") " pod="openshift-dns/node-resolver-d7b2n" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580450 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbf4d\" (UniqueName: \"kubernetes.io/projected/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-kube-api-access-rbf4d\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580498 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5fa87729-4cd3-4453-a444-9c59a6514753-serviceca\") pod \"node-ca-4ptfz\" (UID: \"5fa87729-4cd3-4453-a444-9c59a6514753\") " pod="openshift-image-registry/node-ca-4ptfz" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580553 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-kubelet\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-tuned\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580611 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580762 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-run\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580785 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-cnibin\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580860 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-node-log\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580706 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580821 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-sysctl-d\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580918 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/807cdaf2-aa32-4788-9661-fb1b56a34fc2-ovnkube-script-lib\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580933 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-sysconfig\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.580989 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-etc-openvswitch\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.581122 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22dw\" (UniqueName: \"kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw\") pod \"network-check-target-x2vn2\" (UID: \"458ea307-1c97-4cda-8723-97e047f76a50\") " pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.581137 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-modprobe-d\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.581261 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-cni-bin\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.581706 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.581336 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-run-netns\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.582306 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.581383 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-lib-modules\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.582306 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.581414 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-systemd\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.582306 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.581450 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-log-socket\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.582306 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.581451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-cni-binary-copy\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.582306 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.581334 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782eb993-d0db-4c25-b990-34dbe4d7a5b9-tmp-dir\") pod \"node-resolver-d7b2n\" (UID: \"782eb993-d0db-4c25-b990-34dbe4d7a5b9\") " pod="openshift-dns/node-resolver-d7b2n" Jul 16 17:34:12.582306 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.581611 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-systemd\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.582306 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.581637 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-sysctl-conf\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.582306 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.581774 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-systemd-units\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.582306 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582022 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.582306 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582059 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-cni-netd\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.582306 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582117 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-cni-netd\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.582306 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582274 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.582765 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582361 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5fa87729-4cd3-4453-a444-9c59a6514753-serviceca\") pod \"node-ca-4ptfz\" (UID: \"5fa87729-4cd3-4453-a444-9c59a6514753\") " pod="openshift-image-registry/node-ca-4ptfz" Jul 16 17:34:12.582765 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582420 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-cni-binary-copy\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.582765 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582499 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.582765 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582523 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/807cdaf2-aa32-4788-9661-fb1b56a34fc2-ovnkube-script-lib\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.582765 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ggvm\" (UniqueName: \"kubernetes.io/projected/807cdaf2-aa32-4788-9661-fb1b56a34fc2-kube-api-access-9ggvm\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.582765 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582615 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-host\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.582765 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582620 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/807cdaf2-aa32-4788-9661-fb1b56a34fc2-env-overrides\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.582765 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582675 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-slash\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.582765 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582714 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-system-cni-dir\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.582765 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582705 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.583263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582795 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-os-release\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.583263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582816 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-slash\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.583263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582871 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/807cdaf2-aa32-4788-9661-fb1b56a34fc2-ovn-node-metrics-cert\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.583263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582896 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-os-release\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.583263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5fa87729-4cd3-4453-a444-9c59a6514753-host\") pod \"node-ca-4ptfz\" (UID: \"5fa87729-4cd3-4453-a444-9c59a6514753\") " pod="openshift-image-registry/node-ca-4ptfz" Jul 16 17:34:12.583263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.582987 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-run-systemd\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.583263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583029 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/807cdaf2-aa32-4788-9661-fb1b56a34fc2-env-overrides\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.583263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583075 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-var-lib-openvswitch\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.583263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-run-ovn\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.583263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583173 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-sys\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.583263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583177 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-system-cni-dir\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.583263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgcft\" (UniqueName: \"kubernetes.io/projected/782eb993-d0db-4c25-b990-34dbe4d7a5b9-kube-api-access-cgcft\") pod \"node-resolver-d7b2n\" (UID: \"782eb993-d0db-4c25-b990-34dbe4d7a5b9\") " pod="openshift-dns/node-resolver-d7b2n" Jul 16 17:34:12.583263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583230 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-kubernetes\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.583860 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/782eb993-d0db-4c25-b990-34dbe4d7a5b9-hosts-file\") pod \"node-resolver-d7b2n\" (UID: \"782eb993-d0db-4c25-b990-34dbe4d7a5b9\") " pod="openshift-dns/node-resolver-d7b2n" Jul 16 17:34:12.583860 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583357 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-run-openvswitch\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.583860 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583445 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-var-lib-kubelet\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.583860 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-run-ovn-kubernetes\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.583860 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583538 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-run-ovn\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.583860 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583623 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-sys\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.583860 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583625 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.583860 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583667 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-run-openvswitch\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.583860 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583415 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/782eb993-d0db-4c25-b990-34dbe4d7a5b9-hosts-file\") pod \"node-resolver-d7b2n\" (UID: \"782eb993-d0db-4c25-b990-34dbe4d7a5b9\") " pod="openshift-dns/node-resolver-d7b2n" Jul 16 17:34:12.583860 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.583860 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583808 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-run-ovn-kubernetes\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.583860 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583847 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/807cdaf2-aa32-4788-9661-fb1b56a34fc2-ovnkube-config\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.584398 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnd65\" (UniqueName: \"kubernetes.io/projected/5fa87729-4cd3-4453-a444-9c59a6514753-kube-api-access-nnd65\") pod \"node-ca-4ptfz\" (UID: \"5fa87729-4cd3-4453-a444-9c59a6514753\") " pod="openshift-image-registry/node-ca-4ptfz" Jul 16 17:34:12.584398 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.583968 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52d2c0de-2ebc-45eb-ad30-aad441699edd-var-lib-kubelet\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.584398 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.584269 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-run-systemd\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.584398 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.584366 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-var-lib-openvswitch\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.584605 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.584431 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5fa87729-4cd3-4453-a444-9c59a6514753-host\") pod \"node-ca-4ptfz\" (UID: \"5fa87729-4cd3-4453-a444-9c59a6514753\") " pod="openshift-image-registry/node-ca-4ptfz" Jul 16 17:34:12.584763 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.584718 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/807cdaf2-aa32-4788-9661-fb1b56a34fc2-host-kubelet\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.584900 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.584821 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/807cdaf2-aa32-4788-9661-fb1b56a34fc2-ovnkube-config\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.585971 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.585948 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52d2c0de-2ebc-45eb-ad30-aad441699edd-tmp\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.586118 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.586103 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52d2c0de-2ebc-45eb-ad30-aad441699edd-etc-tuned\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.587524 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.587452 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.ec2.internal" event={"ID":"b8cc93a3e3b0301bf02b2011bdc86d70","Type":"ContainerStarted","Data":"7c30513d0c4cfbbbf9d0cd8e951e1bb1a3e36cb92decba58533a3e38d77bbac5"} Jul 16 17:34:12.591267 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.590639 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:34:12.591267 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.590668 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:34:12.591267 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.590684 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m22dw for pod openshift-network-diagnostics/network-check-target-x2vn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:12.591267 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.590758 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw podName:458ea307-1c97-4cda-8723-97e047f76a50 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:13.090739413 +0000 UTC m=+3.107968747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m22dw" (UniqueName: "kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw") pod "network-check-target-x2vn2" (UID: "458ea307-1c97-4cda-8723-97e047f76a50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:12.592457 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.592355 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/807cdaf2-aa32-4788-9661-fb1b56a34fc2-ovn-node-metrics-cert\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.594740 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.594710 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbf4d\" (UniqueName: \"kubernetes.io/projected/63aefcda-40ba-4e5d-8fd4-491a9bf5b45b-kube-api-access-rbf4d\") pod \"multus-additional-cni-plugins-ct6j5\" (UID: \"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b\") " pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.595602 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.595160 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvhk\" (UniqueName: \"kubernetes.io/projected/52d2c0de-2ebc-45eb-ad30-aad441699edd-kube-api-access-2jvhk\") pod \"tuned-h6xsw\" (UID: \"52d2c0de-2ebc-45eb-ad30-aad441699edd\") " pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.595602 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.595161 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgcft\" (UniqueName: \"kubernetes.io/projected/782eb993-d0db-4c25-b990-34dbe4d7a5b9-kube-api-access-cgcft\") pod \"node-resolver-d7b2n\" (UID: \"782eb993-d0db-4c25-b990-34dbe4d7a5b9\") " pod="openshift-dns/node-resolver-d7b2n" Jul 16 17:34:12.595602 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.595383 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ggvm\" (UniqueName: \"kubernetes.io/projected/807cdaf2-aa32-4788-9661-fb1b56a34fc2-kube-api-access-9ggvm\") pod \"ovnkube-node-4lfqt\" (UID: \"807cdaf2-aa32-4788-9661-fb1b56a34fc2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.595602 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.595556 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnd65\" (UniqueName: \"kubernetes.io/projected/5fa87729-4cd3-4453-a444-9c59a6514753-kube-api-access-nnd65\") pod \"node-ca-4ptfz\" (UID: \"5fa87729-4cd3-4453-a444-9c59a6514753\") " pod="openshift-image-registry/node-ca-4ptfz" Jul 16 17:34:12.680500 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.680444 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-855sl" Jul 16 17:34:12.689011 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.688983 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7kc7q" Jul 16 17:34:12.690254 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:12.690222 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e92fef_28b7_4d9e_af27_d22d54ca66a6.slice/crio-a1a5dbfd82eba5b0772891be07f779e6c8bcad450ac45cade70d9c2aa722657c WatchSource:0}: Error finding container a1a5dbfd82eba5b0772891be07f779e6c8bcad450ac45cade70d9c2aa722657c: Status 404 returned error can't find the container with id a1a5dbfd82eba5b0772891be07f779e6c8bcad450ac45cade70d9c2aa722657c Jul 16 17:34:12.698046 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.698014 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" Jul 16 17:34:12.698379 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:12.698351 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30219802_706a_47e2_9bf1_8942aade426c.slice/crio-37f979db96293e1d3fe20f6d34b42a0015ba8c74a300fa424053c62dd1f04c78 WatchSource:0}: Error finding container 37f979db96293e1d3fe20f6d34b42a0015ba8c74a300fa424053c62dd1f04c78: Status 404 returned error can't find the container with id 37f979db96293e1d3fe20f6d34b42a0015ba8c74a300fa424053c62dd1f04c78 Jul 16 17:34:12.703419 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.703395 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qj6ls" Jul 16 17:34:12.708897 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:12.708861 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728b9423_5d3a_471d_8eb6_f23a19614cd0.slice/crio-42139144843a6927f43e19f666e5398a6ef7bba267fd7bcf6db34c036c5886dd WatchSource:0}: Error finding container 42139144843a6927f43e19f666e5398a6ef7bba267fd7bcf6db34c036c5886dd: Status 404 returned error can't find the container with id 42139144843a6927f43e19f666e5398a6ef7bba267fd7bcf6db34c036c5886dd Jul 16 17:34:12.711384 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.711357 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:12.715142 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:12.715119 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59c2366_fc5e_4e31_a94f_5a39c99dc3d5.slice/crio-0b25d097321fec54195bb7bd8a8d0ff39412016ad944646113b60a5534b79648 WatchSource:0}: Error finding container 0b25d097321fec54195bb7bd8a8d0ff39412016ad944646113b60a5534b79648: Status 404 returned error can't find the container with id 0b25d097321fec54195bb7bd8a8d0ff39412016ad944646113b60a5534b79648 Jul 16 17:34:12.720527 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.720459 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" Jul 16 17:34:12.727687 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.727654 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d7b2n" Jul 16 17:34:12.735458 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.735423 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4ptfz" Jul 16 17:34:12.741293 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.741269 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ct6j5" Jul 16 17:34:12.987628 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.987594 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:12.987801 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:12.987657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:12.987863 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.987790 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:12.987901 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.987872 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret podName:503d02a0-3e8a-4786-bf7e-c4c0bec8dbda nodeName:}" failed. No retries permitted until 2026-07-16 17:34:13.987851326 +0000 UTC m=+4.005080651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret") pod "global-pull-secret-syncer-hfrlv" (UID: "503d02a0-3e8a-4786-bf7e-c4c0bec8dbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:12.987901 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.987792 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:12.987998 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:12.987940 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs podName:dc349e33-d51c-40bf-8890-392c3393299d nodeName:}" failed. No retries permitted until 2026-07-16 17:34:13.987917909 +0000 UTC m=+4.005147229 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs") pod "network-metrics-daemon-5qrcw" (UID: "dc349e33-d51c-40bf-8890-392c3393299d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:13.189549 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.189416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22dw\" (UniqueName: \"kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw\") pod \"network-check-target-x2vn2\" (UID: \"458ea307-1c97-4cda-8723-97e047f76a50\") " pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:13.189727 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:13.189610 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:34:13.189727 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:13.189636 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:34:13.189727 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:13.189650 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m22dw for pod openshift-network-diagnostics/network-check-target-x2vn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:13.189727 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:13.189720 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw podName:458ea307-1c97-4cda-8723-97e047f76a50 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:14.189700606 +0000 UTC m=+4.206929932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m22dw" (UniqueName: "kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw") pod "network-check-target-x2vn2" (UID: "458ea307-1c97-4cda-8723-97e047f76a50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:13.404374 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:13.404215 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807cdaf2_aa32_4788_9661_fb1b56a34fc2.slice/crio-814fdffe89a34d842b93498d41fbfa5059d5ed77b70ac91fa2815087c0e2dca1 WatchSource:0}: Error finding container 814fdffe89a34d842b93498d41fbfa5059d5ed77b70ac91fa2815087c0e2dca1: Status 404 returned error can't find the container with id 814fdffe89a34d842b93498d41fbfa5059d5ed77b70ac91fa2815087c0e2dca1 Jul 16 17:34:13.405704 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:13.405672 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782eb993_d0db_4c25_b990_34dbe4d7a5b9.slice/crio-c3e3aa22c682b4301c8c32cea1d8dd7623c69601964b1d658595d4f068c201dd WatchSource:0}: Error finding container c3e3aa22c682b4301c8c32cea1d8dd7623c69601964b1d658595d4f068c201dd: Status 404 returned error can't find the container with id c3e3aa22c682b4301c8c32cea1d8dd7623c69601964b1d658595d4f068c201dd Jul 16 17:34:13.406485 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:13.406449 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa87729_4cd3_4453_a444_9c59a6514753.slice/crio-09a0c28281ee1a07d9d9cd20e1df2604ee3dd69203d7ca8c9e25d89af68a3d0f WatchSource:0}: Error finding container 09a0c28281ee1a07d9d9cd20e1df2604ee3dd69203d7ca8c9e25d89af68a3d0f: Status 404 returned error can't find the container with id 09a0c28281ee1a07d9d9cd20e1df2604ee3dd69203d7ca8c9e25d89af68a3d0f Jul 16 17:34:13.410443 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:13.410421 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d2c0de_2ebc_45eb_ad30_aad441699edd.slice/crio-e2b9bd637555a5e64e8cc41109490e79179670677c26f5aa062e974d6f6e73e6 WatchSource:0}: Error finding container e2b9bd637555a5e64e8cc41109490e79179670677c26f5aa062e974d6f6e73e6: Status 404 returned error can't find the container with id e2b9bd637555a5e64e8cc41109490e79179670677c26f5aa062e974d6f6e73e6 Jul 16 17:34:13.412010 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:13.411990 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63aefcda_40ba_4e5d_8fd4_491a9bf5b45b.slice/crio-84718d6258c4edab955948fea509336f71bb46f9384fa06b2764fad7e44ff732 WatchSource:0}: Error finding container 84718d6258c4edab955948fea509336f71bb46f9384fa06b2764fad7e44ff732: Status 404 returned error can't find the container with id 84718d6258c4edab955948fea509336f71bb46f9384fa06b2764fad7e44ff732 Jul 16 17:34:13.419818 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:13.419784 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510a81c26cef2cf6d90419e4745d7adc.slice/crio-ca182d62427488415ab3b958bf291f62fe069a06a37f4eb08e13ba7b9a5c41ac WatchSource:0}: Error finding container ca182d62427488415ab3b958bf291f62fe069a06a37f4eb08e13ba7b9a5c41ac: Status 404 returned error can't find the container with id ca182d62427488415ab3b958bf291f62fe069a06a37f4eb08e13ba7b9a5c41ac Jul 16 17:34:13.436194 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.436152 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:29:12 +0000 UTC" deadline="2028-01-13 15:29:22.134186703 +0000 UTC" Jul 16 17:34:13.436194 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.436189 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13101h55m8.698000902s" Jul 16 17:34:13.594163 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.594121 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.ec2.internal" event={"ID":"b8cc93a3e3b0301bf02b2011bdc86d70","Type":"ContainerStarted","Data":"95f730349c4f7e80a09a4a1241201aa7cd7a3cb29bd67c867d405294ffa3a5a4"} Jul 16 17:34:13.596253 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.596216 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal" event={"ID":"510a81c26cef2cf6d90419e4745d7adc","Type":"ContainerStarted","Data":"ca182d62427488415ab3b958bf291f62fe069a06a37f4eb08e13ba7b9a5c41ac"} Jul 16 17:34:13.598058 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.597996 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct6j5" event={"ID":"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b","Type":"ContainerStarted","Data":"84718d6258c4edab955948fea509336f71bb46f9384fa06b2764fad7e44ff732"} Jul 16 17:34:13.599334 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.599310 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" event={"ID":"52d2c0de-2ebc-45eb-ad30-aad441699edd","Type":"ContainerStarted","Data":"e2b9bd637555a5e64e8cc41109490e79179670677c26f5aa062e974d6f6e73e6"} Jul 16 17:34:13.601299 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.601249 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4ptfz" event={"ID":"5fa87729-4cd3-4453-a444-9c59a6514753","Type":"ContainerStarted","Data":"09a0c28281ee1a07d9d9cd20e1df2604ee3dd69203d7ca8c9e25d89af68a3d0f"} Jul 16 17:34:13.602891 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.602787 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" event={"ID":"807cdaf2-aa32-4788-9661-fb1b56a34fc2","Type":"ContainerStarted","Data":"814fdffe89a34d842b93498d41fbfa5059d5ed77b70ac91fa2815087c0e2dca1"} Jul 16 17:34:13.604337 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.604307 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7kc7q" event={"ID":"30219802-706a-47e2-9bf1-8942aade426c","Type":"ContainerStarted","Data":"37f979db96293e1d3fe20f6d34b42a0015ba8c74a300fa424053c62dd1f04c78"} Jul 16 17:34:13.605644 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.605616 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-855sl" event={"ID":"e4e92fef-28b7-4d9e-af27-d22d54ca66a6","Type":"ContainerStarted","Data":"a1a5dbfd82eba5b0772891be07f779e6c8bcad450ac45cade70d9c2aa722657c"} Jul 16 17:34:13.607678 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.607645 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d7b2n" event={"ID":"782eb993-d0db-4c25-b990-34dbe4d7a5b9","Type":"ContainerStarted","Data":"c3e3aa22c682b4301c8c32cea1d8dd7623c69601964b1d658595d4f068c201dd"} Jul 16 17:34:13.609725 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.609697 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qj6ls" event={"ID":"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5","Type":"ContainerStarted","Data":"0b25d097321fec54195bb7bd8a8d0ff39412016ad944646113b60a5534b79648"} Jul 16 17:34:13.610440 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.610296 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.ec2.internal" podStartSLOduration=2.610172831 podStartE2EDuration="2.610172831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:13.609292615 +0000 UTC m=+3.626521958" watchObservedRunningTime="2026-07-16 17:34:13.610172831 +0000 UTC m=+3.627402174" Jul 16 17:34:13.611582 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.611553 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" event={"ID":"728b9423-5d3a-471d-8eb6-f23a19614cd0","Type":"ContainerStarted","Data":"42139144843a6927f43e19f666e5398a6ef7bba267fd7bcf6db34c036c5886dd"} Jul 16 17:34:13.995895 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.995865 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:13.996079 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:13.995915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:13.996079 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:13.996015 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:13.996151 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:13.996068 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs podName:dc349e33-d51c-40bf-8890-392c3393299d nodeName:}" failed. No retries permitted until 2026-07-16 17:34:15.996052575 +0000 UTC m=+6.013281894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs") pod "network-metrics-daemon-5qrcw" (UID: "dc349e33-d51c-40bf-8890-392c3393299d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:13.996151 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:13.996131 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:13.996151 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:13.996153 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret podName:503d02a0-3e8a-4786-bf7e-c4c0bec8dbda nodeName:}" failed. No retries permitted until 2026-07-16 17:34:15.996146367 +0000 UTC m=+6.013375687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret") pod "global-pull-secret-syncer-hfrlv" (UID: "503d02a0-3e8a-4786-bf7e-c4c0bec8dbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:14.197614 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:14.197574 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22dw\" (UniqueName: \"kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw\") pod \"network-check-target-x2vn2\" (UID: \"458ea307-1c97-4cda-8723-97e047f76a50\") " pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:14.197809 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:14.197785 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:34:14.197809 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:14.197807 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:34:14.197917 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:14.197819 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m22dw for pod openshift-network-diagnostics/network-check-target-x2vn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:14.197917 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:14.197881 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw podName:458ea307-1c97-4cda-8723-97e047f76a50 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:16.197861755 +0000 UTC m=+6.215091083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m22dw" (UniqueName: "kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw") pod "network-check-target-x2vn2" (UID: "458ea307-1c97-4cda-8723-97e047f76a50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:14.436664 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:14.436572 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:29:12 +0000 UTC" deadline="2028-02-01 02:14:39.116284349 +0000 UTC" Jul 16 17:34:14.436664 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:14.436610 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13544h40m24.679677407s" Jul 16 17:34:14.580465 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:14.580430 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:14.580465 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:14.580466 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:14.580687 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:14.580466 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:14.580687 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:14.580605 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:14.580764 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:14.580731 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:14.580866 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:14.580838 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:16.013600 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:16.013552 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:16.014050 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:16.013639 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:16.014050 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:16.013775 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:16.014050 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:16.013836 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret podName:503d02a0-3e8a-4786-bf7e-c4c0bec8dbda nodeName:}" failed. No retries permitted until 2026-07-16 17:34:20.013819148 +0000 UTC m=+10.031048485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret") pod "global-pull-secret-syncer-hfrlv" (UID: "503d02a0-3e8a-4786-bf7e-c4c0bec8dbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:16.014274 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:16.014257 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:16.014349 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:16.014311 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs podName:dc349e33-d51c-40bf-8890-392c3393299d nodeName:}" failed. No retries permitted until 2026-07-16 17:34:20.014295108 +0000 UTC m=+10.031524430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs") pod "network-metrics-daemon-5qrcw" (UID: "dc349e33-d51c-40bf-8890-392c3393299d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:16.215634 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:16.215596 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22dw\" (UniqueName: \"kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw\") pod \"network-check-target-x2vn2\" (UID: \"458ea307-1c97-4cda-8723-97e047f76a50\") " pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:16.215813 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:16.215785 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:34:16.215813 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:16.215803 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:34:16.215894 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:16.215816 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m22dw for pod openshift-network-diagnostics/network-check-target-x2vn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:16.215894 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:16.215871 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw podName:458ea307-1c97-4cda-8723-97e047f76a50 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:20.215852907 +0000 UTC m=+10.233082233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m22dw" (UniqueName: "kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw") pod "network-check-target-x2vn2" (UID: "458ea307-1c97-4cda-8723-97e047f76a50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:16.582490 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:16.582439 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:16.582677 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:16.582584 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:16.583002 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:16.582982 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:16.583104 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:16.583084 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:16.583167 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:16.583138 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:16.583217 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:16.583193 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:18.579889 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:18.579802 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:18.579889 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:18.579840 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:18.580431 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:18.579800 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:18.580431 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:18.579946 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:18.580431 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:18.580034 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:18.580431 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:18.580112 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:20.046597 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:20.046554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:20.047048 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:20.046621 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:20.047048 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:20.046737 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:20.047048 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:20.046737 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:20.047048 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:20.046805 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs podName:dc349e33-d51c-40bf-8890-392c3393299d nodeName:}" failed. No retries permitted until 2026-07-16 17:34:28.046788373 +0000 UTC m=+18.064017700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs") pod "network-metrics-daemon-5qrcw" (UID: "dc349e33-d51c-40bf-8890-392c3393299d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:20.047048 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:20.046824 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret podName:503d02a0-3e8a-4786-bf7e-c4c0bec8dbda nodeName:}" failed. No retries permitted until 2026-07-16 17:34:28.046814279 +0000 UTC m=+18.064043609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret") pod "global-pull-secret-syncer-hfrlv" (UID: "503d02a0-3e8a-4786-bf7e-c4c0bec8dbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:20.247324 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:20.247288 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22dw\" (UniqueName: \"kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw\") pod \"network-check-target-x2vn2\" (UID: \"458ea307-1c97-4cda-8723-97e047f76a50\") " pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:20.247531 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:20.247493 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:34:20.247531 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:20.247519 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:34:20.247531 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:20.247532 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m22dw for pod openshift-network-diagnostics/network-check-target-x2vn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:20.247697 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:20.247593 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw podName:458ea307-1c97-4cda-8723-97e047f76a50 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:28.247573262 +0000 UTC m=+18.264802599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m22dw" (UniqueName: "kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw") pod "network-check-target-x2vn2" (UID: "458ea307-1c97-4cda-8723-97e047f76a50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:20.581495 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:20.580880 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:20.581495 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:20.580970 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:20.581495 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:20.581279 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:20.581495 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:20.581346 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:20.581495 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:20.581397 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:20.581495 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:20.581446 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:22.581897 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:22.581870 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:22.582323 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:22.581983 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:22.582403 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:22.582359 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:22.582487 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:22.582451 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:22.582556 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:22.582530 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:22.582642 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:22.582622 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:24.580236 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:24.580201 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:24.580658 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:24.580325 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:24.580658 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:24.580576 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:24.580754 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:24.580697 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:24.580754 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:24.580747 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:24.580846 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:24.580828 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:26.579975 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:26.579930 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:26.579975 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:26.579976 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:26.580511 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:26.579943 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:26.580511 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:26.580054 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:26.580511 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:26.580158 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:26.580511 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:26.580248 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:28.103340 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:28.103258 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:28.103340 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:28.103315 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:28.103873 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:28.103408 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:28.103873 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:28.103459 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs podName:dc349e33-d51c-40bf-8890-392c3393299d nodeName:}" failed. No retries permitted until 2026-07-16 17:34:44.103446354 +0000 UTC m=+34.120675678 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs") pod "network-metrics-daemon-5qrcw" (UID: "dc349e33-d51c-40bf-8890-392c3393299d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:28.103873 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:28.103408 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:28.103873 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:28.103582 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret podName:503d02a0-3e8a-4786-bf7e-c4c0bec8dbda nodeName:}" failed. No retries permitted until 2026-07-16 17:34:44.103560653 +0000 UTC m=+34.120789995 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret") pod "global-pull-secret-syncer-hfrlv" (UID: "503d02a0-3e8a-4786-bf7e-c4c0bec8dbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:28.304761 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:28.304723 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22dw\" (UniqueName: \"kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw\") pod \"network-check-target-x2vn2\" (UID: \"458ea307-1c97-4cda-8723-97e047f76a50\") " pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:28.304928 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:28.304901 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:34:28.304928 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:28.304924 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:34:28.305026 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:28.304934 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m22dw for pod openshift-network-diagnostics/network-check-target-x2vn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:28.305026 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:28.304988 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw podName:458ea307-1c97-4cda-8723-97e047f76a50 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:44.304969954 +0000 UTC m=+34.322199287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m22dw" (UniqueName: "kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw") pod "network-check-target-x2vn2" (UID: "458ea307-1c97-4cda-8723-97e047f76a50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:28.580897 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:28.580484 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:28.580897 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:28.580599 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:28.580897 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:28.580495 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:28.580897 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:28.580626 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:28.580897 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:28.580727 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:28.580897 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:28.580825 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:30.580077 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:30.580042 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:30.580548 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:30.580151 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:30.580548 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:30.580169 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:30.580548 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:30.580251 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:30.580548 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:30.580304 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:30.580548 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:30.580380 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:32.580095 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.579754 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:32.580779 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.579754 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:32.580779 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:32.580170 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:32.580779 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:32.580278 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:32.580779 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.579813 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:32.580779 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:32.580393 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:32.650922 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.650895 2582 generic.go:356] "Generic (PLEG): container finished" podID="510a81c26cef2cf6d90419e4745d7adc" containerID="cfe3d95fae5f0f46b67282c6a95eb84d3708d5160bbb58ac65a6b119821cdd8e" exitCode=0 Jul 16 17:34:32.651073 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.650960 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal" event={"ID":"510a81c26cef2cf6d90419e4745d7adc","Type":"ContainerDied","Data":"cfe3d95fae5f0f46b67282c6a95eb84d3708d5160bbb58ac65a6b119821cdd8e"} Jul 16 17:34:32.652701 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.652679 2582 generic.go:356] "Generic (PLEG): container finished" podID="63aefcda-40ba-4e5d-8fd4-491a9bf5b45b" containerID="f4e2b22db5f8d3603753a0db0f2f08e1bef749b3b9183e26a645ef4523048c1d" exitCode=0 Jul 16 17:34:32.652856 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.652710 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct6j5" event={"ID":"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b","Type":"ContainerDied","Data":"f4e2b22db5f8d3603753a0db0f2f08e1bef749b3b9183e26a645ef4523048c1d"} Jul 16 17:34:32.654531 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.654503 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" event={"ID":"52d2c0de-2ebc-45eb-ad30-aad441699edd","Type":"ContainerStarted","Data":"b2acff88c926f8b4fbbe2a011cbb2ec769da10ff7b484456aae88715c21fcc9f"} Jul 16 17:34:32.656705 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.656675 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4ptfz" event={"ID":"5fa87729-4cd3-4453-a444-9c59a6514753","Type":"ContainerStarted","Data":"44d607ac2d1ece12f4410b81992e4fcced12a25b60bfd48e046677500b2c39d6"} Jul 16 17:34:32.661241 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.661226 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 17:34:32.661590 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.661574 2582 generic.go:356] "Generic (PLEG): container finished" podID="807cdaf2-aa32-4788-9661-fb1b56a34fc2" containerID="27b52721e1fe926f804dece541c0d60f20fd9a55ac910309d508ea66a8189c08" exitCode=1 Jul 16 17:34:32.661657 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.661639 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" event={"ID":"807cdaf2-aa32-4788-9661-fb1b56a34fc2","Type":"ContainerStarted","Data":"076fbe4ef023b45f5e9d034d7ce22c9f8999fd68102f24573d938ae25eac4d4a"} Jul 16 17:34:32.661709 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.661697 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" event={"ID":"807cdaf2-aa32-4788-9661-fb1b56a34fc2","Type":"ContainerStarted","Data":"555ae8e45e3a867f5ad12bbb0351da133821431dad4c415754182ab5ccc25d46"} Jul 16 17:34:32.661757 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.661746 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" event={"ID":"807cdaf2-aa32-4788-9661-fb1b56a34fc2","Type":"ContainerStarted","Data":"d6d198302de30e8c879eda03142345f5b0956e2c0f4dd179d481870023328f03"} Jul 16 17:34:32.661800 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.661786 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" event={"ID":"807cdaf2-aa32-4788-9661-fb1b56a34fc2","Type":"ContainerStarted","Data":"4acec1b0db7dd5d8a91441914c7d8d6a7de6afce1dbe7fe7136e2faff304d283"} Jul 16 17:34:32.661841 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.661829 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" event={"ID":"807cdaf2-aa32-4788-9661-fb1b56a34fc2","Type":"ContainerDied","Data":"27b52721e1fe926f804dece541c0d60f20fd9a55ac910309d508ea66a8189c08"} Jul 16 17:34:32.661879 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.661868 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" event={"ID":"807cdaf2-aa32-4788-9661-fb1b56a34fc2","Type":"ContainerStarted","Data":"1d9f34b7136406af33e7b024d4830c3f547877db32dabb8351408e01798d4453"} Jul 16 17:34:32.663049 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.663028 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7kc7q" event={"ID":"30219802-706a-47e2-9bf1-8942aade426c","Type":"ContainerStarted","Data":"ea7d9eb748d2e3e81025fe43b5c2c9ffeb18a56c67fdf95ac183301ed670e7c3"} Jul 16 17:34:32.664263 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.664241 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d7b2n" event={"ID":"782eb993-d0db-4c25-b990-34dbe4d7a5b9","Type":"ContainerStarted","Data":"9f42ec48c8a80bbdf89f700d0c64adbb6645d3b5c6d48558cc05e87d0271f386"} Jul 16 17:34:32.665724 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.665698 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qj6ls" event={"ID":"e59c2366-fc5e-4e31-a94f-5a39c99dc3d5","Type":"ContainerStarted","Data":"b1e69f6e241a599c92ec9ca80c91c466e6cbddb270799c2173ec79ee4cddc98e"} Jul 16 17:34:32.667326 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.667303 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" event={"ID":"728b9423-5d3a-471d-8eb6-f23a19614cd0","Type":"ContainerStarted","Data":"3fbb31f1c93c49e998ec5e964ba7c4860363a1551382ba6901e504099c889569"} Jul 16 17:34:32.703795 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.703759 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qj6ls" podStartSLOduration=3.6969364110000003 podStartE2EDuration="22.703746465s" totalImagesPullingTime="19.006810054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:32.703246677 +0000 UTC m=+22.720476039" watchObservedRunningTime="2026-07-16 17:34:32.703746465 +0000 UTC m=+22.720975831" Jul 16 17:34:32.718748 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.718707 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h6xsw" podStartSLOduration=4.52467377 podStartE2EDuration="22.718696519s" totalImagesPullingTime="18.194022749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:32.718593106 +0000 UTC m=+22.735822475" watchObservedRunningTime="2026-07-16 17:34:32.718696519 +0000 UTC m=+22.735925861" Jul 16 17:34:32.732462 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.732425 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7kc7q" podStartSLOduration=12.842757755000001 podStartE2EDuration="22.732415262s" totalImagesPullingTime="9.889657507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:32.732006811 +0000 UTC m=+22.749236156" watchObservedRunningTime="2026-07-16 17:34:32.732415262 +0000 UTC m=+22.749644604" Jul 16 17:34:32.747729 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.747695 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d7b2n" podStartSLOduration=4.549139916 podStartE2EDuration="22.747684136s" totalImagesPullingTime="18.19854422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:32.747288181 +0000 UTC m=+22.764517513" watchObservedRunningTime="2026-07-16 17:34:32.747684136 +0000 UTC m=+22.764913478" Jul 16 17:34:32.763128 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:32.763093 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4ptfz" podStartSLOduration=4.499316934 podStartE2EDuration="22.763081812s" totalImagesPullingTime="18.263764878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:32.76264793 +0000 UTC m=+22.779877274" watchObservedRunningTime="2026-07-16 17:34:32.763081812 +0000 UTC m=+22.780311154" Jul 16 17:34:33.244744 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:33.244715 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 17:34:33.436548 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:33.436418 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T17:34:33.244738736Z","UUID":"c65aaab0-24c6-4779-b504-a0b84c0b1dd0","Handler":null,"Name":"","Endpoint":""} Jul 16 17:34:33.437774 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:33.437749 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 17:34:33.437863 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:33.437785 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 17:34:33.672004 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:33.671972 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal" event={"ID":"510a81c26cef2cf6d90419e4745d7adc","Type":"ContainerStarted","Data":"ddb72a56ac5bb62e719b5f70477b4462a2b4c4477a11dfaa51021bd8dc5f53f4"} Jul 16 17:34:33.673685 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:33.673654 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-855sl" event={"ID":"e4e92fef-28b7-4d9e-af27-d22d54ca66a6","Type":"ContainerStarted","Data":"583a84ba5e7226b1af3eacee6da17a4cef32c92431be9c2ac3fa34eb2fa15962"} Jul 16 17:34:33.675743 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:33.675717 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" event={"ID":"728b9423-5d3a-471d-8eb6-f23a19614cd0","Type":"ContainerStarted","Data":"3f20d618cadd7dd73873c1fb37b975cd84633240b63431b99ef0376aeb36d214"} Jul 16 17:34:33.688227 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:33.688141 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.ec2.internal" podStartSLOduration=22.688125624 podStartE2EDuration="22.688125624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:33.687750976 +0000 UTC m=+23.704980317" watchObservedRunningTime="2026-07-16 17:34:33.688125624 +0000 UTC m=+23.705354965" Jul 16 17:34:34.579864 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:34.579780 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:34.579864 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:34.579837 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:34.579864 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:34.579842 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:34.580114 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:34.579942 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:34.580166 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:34.580117 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:34.580254 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:34.580222 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:34.680188 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:34.680158 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" event={"ID":"728b9423-5d3a-471d-8eb6-f23a19614cd0","Type":"ContainerStarted","Data":"be5fa4906a3768d1040702a785b456b955c4a93bffa2e7f159e00f63b16f7e64"} Jul 16 17:34:34.683941 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:34.683914 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 17:34:34.684487 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:34.684439 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" event={"ID":"807cdaf2-aa32-4788-9661-fb1b56a34fc2","Type":"ContainerStarted","Data":"4efb5670a1b69ae0e047d5f5c147c9db14f19ef694136f452e02dcd50e3a9d45"} Jul 16 17:34:34.699788 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:34.699750 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-855sl" podStartSLOduration=14.024909212 podStartE2EDuration="24.699737028s" totalImagesPullingTime="10.674827816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:33.701725406 +0000 UTC m=+23.718954748" watchObservedRunningTime="2026-07-16 17:34:34.699737028 +0000 UTC m=+24.716966372" Jul 16 17:34:34.700074 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:34.700046 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ql6hb" podStartSLOduration=7.848490123 podStartE2EDuration="24.700039943s" totalImagesPullingTime="16.85154982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:34.69924738 +0000 UTC m=+24.716476723" watchObservedRunningTime="2026-07-16 17:34:34.700039943 +0000 UTC m=+24.717269285" Jul 16 17:34:35.335034 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:35.334847 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7kc7q" Jul 16 17:34:35.335411 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:35.335390 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7kc7q" Jul 16 17:34:36.579529 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:36.579491 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:36.580171 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:36.579609 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:36.580171 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:36.579707 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:36.580171 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:36.579814 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:36.580171 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:36.579878 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:36.580171 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:36.579966 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:36.691652 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:36.691593 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 17:34:36.691960 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:36.691931 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" event={"ID":"807cdaf2-aa32-4788-9661-fb1b56a34fc2","Type":"ContainerStarted","Data":"d562e2c5b67687e9a83cc0868803df29feda6a3a5c337b4c64219b374b0eb1ae"} Jul 16 17:34:36.692216 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:36.692194 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:36.692319 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:36.692277 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:36.692379 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:36.692327 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:36.692430 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:36.692415 2582 scope.go:117] "RemoveContainer" containerID="27b52721e1fe926f804dece541c0d60f20fd9a55ac910309d508ea66a8189c08" Jul 16 17:34:36.708144 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:36.708118 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:36.708611 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:36.708595 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:34:37.707660 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:37.707360 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 17:34:37.710285 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:37.709120 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" event={"ID":"807cdaf2-aa32-4788-9661-fb1b56a34fc2","Type":"ContainerStarted","Data":"65c1a571545332ba46d336c4ec22a2f3d7a9d82352586add1b1468753ef6f4e5"} Jul 16 17:34:37.743817 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:37.743759 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" podStartSLOduration=9.461825403 podStartE2EDuration="27.743738981s" totalImagesPullingTime="18.281913578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:37.743318932 +0000 UTC m=+27.760548299" watchObservedRunningTime="2026-07-16 17:34:37.743738981 +0000 UTC m=+27.760968325" Jul 16 17:34:38.571397 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:38.571372 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x2vn2"] Jul 16 17:34:38.571569 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:38.571515 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:38.571617 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:38.571593 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:38.573920 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:38.573896 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hfrlv"] Jul 16 17:34:38.574060 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:38.574023 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:38.574130 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:38.574106 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:38.574756 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:38.574731 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qrcw"] Jul 16 17:34:38.574868 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:38.574837 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:38.574959 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:38.574938 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:39.618540 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:39.618512 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7kc7q" Jul 16 17:34:39.618980 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:39.618609 2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 16 17:34:39.619076 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:39.619062 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7kc7q" Jul 16 17:34:39.715957 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:39.715933 2582 generic.go:356] "Generic (PLEG): container finished" podID="63aefcda-40ba-4e5d-8fd4-491a9bf5b45b" containerID="7caa851fdc630a21d2cd3da9b936936776146b02aadec31c1e139c3eb75652c0" exitCode=0 Jul 16 17:34:39.716089 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:39.716015 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct6j5" event={"ID":"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b","Type":"ContainerDied","Data":"7caa851fdc630a21d2cd3da9b936936776146b02aadec31c1e139c3eb75652c0"} Jul 16 17:34:40.580888 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:40.580859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:40.580993 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:40.580957 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:40.581041 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:40.580973 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:40.581089 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:40.581053 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:40.581138 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:40.581082 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:40.581222 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:40.581203 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:42.579518 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:42.579457 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:42.579518 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:42.579488 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:42.580093 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:42.579594 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfrlv" podUID="503d02a0-3e8a-4786-bf7e-c4c0bec8dbda" Jul 16 17:34:42.580093 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:42.579637 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2vn2" podUID="458ea307-1c97-4cda-8723-97e047f76a50" Jul 16 17:34:42.580093 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:42.579675 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:42.580093 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:42.579774 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qrcw" podUID="dc349e33-d51c-40bf-8890-392c3393299d" Jul 16 17:34:42.723614 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:42.723580 2582 generic.go:356] "Generic (PLEG): container finished" podID="63aefcda-40ba-4e5d-8fd4-491a9bf5b45b" containerID="e0f0ffd0bd043a65756d6294b555cd612c147427dd96aed9eaaccd8d025393b5" exitCode=0 Jul 16 17:34:42.723724 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:42.723637 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct6j5" event={"ID":"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b","Type":"ContainerDied","Data":"e0f0ffd0bd043a65756d6294b555cd612c147427dd96aed9eaaccd8d025393b5"} Jul 16 17:34:44.121986 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.121923 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:44.121986 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.121971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:44.122317 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:44.122056 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:44.122317 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:44.122058 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:44.122317 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:44.122102 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs podName:dc349e33-d51c-40bf-8890-392c3393299d nodeName:}" failed. No retries permitted until 2026-07-16 17:35:16.122087905 +0000 UTC m=+66.139317227 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs") pod "network-metrics-daemon-5qrcw" (UID: "dc349e33-d51c-40bf-8890-392c3393299d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:44.122317 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:44.122113 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret podName:503d02a0-3e8a-4786-bf7e-c4c0bec8dbda nodeName:}" failed. No retries permitted until 2026-07-16 17:35:16.122107702 +0000 UTC m=+66.139337022 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret") pod "global-pull-secret-syncer-hfrlv" (UID: "503d02a0-3e8a-4786-bf7e-c4c0bec8dbda") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:44.298236 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.298214 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-103.ec2.internal" event="NodeReady" Jul 16 17:34:44.298322 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.298306 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 17:34:44.323516 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.323493 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22dw\" (UniqueName: \"kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw\") pod \"network-check-target-x2vn2\" (UID: \"458ea307-1c97-4cda-8723-97e047f76a50\") " pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:44.323617 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:44.323604 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:34:44.323654 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:44.323621 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:34:44.323654 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:44.323630 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m22dw for pod openshift-network-diagnostics/network-check-target-x2vn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:44.323718 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:44.323672 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw podName:458ea307-1c97-4cda-8723-97e047f76a50 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:16.32365975 +0000 UTC m=+66.340889072 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m22dw" (UniqueName: "kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw") pod "network-check-target-x2vn2" (UID: "458ea307-1c97-4cda-8723-97e047f76a50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:44.348020 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.347999 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-85spg"] Jul 16 17:34:44.363264 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.363243 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-85spg" Jul 16 17:34:44.363264 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.363253 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5v4vt"] Jul 16 17:34:44.365439 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.365410 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 17:34:44.365439 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.365435 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ntp5m\"" Jul 16 17:34:44.365615 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.365529 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 17:34:44.387442 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.387387 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-85spg"] Jul 16 17:34:44.387442 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.387418 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5v4vt"] Jul 16 17:34:44.387627 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.387527 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5v4vt" Jul 16 17:34:44.389798 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.389779 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 17:34:44.389906 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.389893 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 17:34:44.389973 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.389907 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 17:34:44.389973 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.389932 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n568h\"" Jul 16 17:34:44.524718 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.524693 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjnh9\" (UniqueName: \"kubernetes.io/projected/142b45fe-5371-4dc2-998b-a4aedafa976c-kube-api-access-zjnh9\") pod \"ingress-canary-5v4vt\" (UID: \"142b45fe-5371-4dc2-998b-a4aedafa976c\") " pod="openshift-ingress-canary/ingress-canary-5v4vt" Jul 16 17:34:44.524821 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.524732 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:44.524821 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.524759 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/142b45fe-5371-4dc2-998b-a4aedafa976c-cert\") pod \"ingress-canary-5v4vt\" (UID: \"142b45fe-5371-4dc2-998b-a4aedafa976c\") " pod="openshift-ingress-canary/ingress-canary-5v4vt" Jul 16 17:34:44.524913 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.524830 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f59e0fc2-4364-4718-9c16-34bf2c019278-config-volume\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:44.524913 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.524851 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59e0fc2-4364-4718-9c16-34bf2c019278-tmp-dir\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:44.524913 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.524879 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g7lj\" (UniqueName: \"kubernetes.io/projected/f59e0fc2-4364-4718-9c16-34bf2c019278-kube-api-access-2g7lj\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:44.579907 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.579882 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:34:44.579991 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.579882 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:34:44.580069 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.579892 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:34:44.583036 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.582872 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 17:34:44.583036 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.582883 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s9dx5\"" Jul 16 17:34:44.583036 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.582936 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 17:34:44.583256 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.583240 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 17:34:44.583321 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.583255 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 17:34:44.583373 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.583362 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p9hcv\"" Jul 16 17:34:44.626001 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.625984 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:44.626095 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.626008 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/142b45fe-5371-4dc2-998b-a4aedafa976c-cert\") pod \"ingress-canary-5v4vt\" (UID: \"142b45fe-5371-4dc2-998b-a4aedafa976c\") " pod="openshift-ingress-canary/ingress-canary-5v4vt" Jul 16 17:34:44.626095 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.626030 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f59e0fc2-4364-4718-9c16-34bf2c019278-config-volume\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:44.626095 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.626046 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59e0fc2-4364-4718-9c16-34bf2c019278-tmp-dir\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:44.626250 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:44.626158 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:34:44.626250 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.626170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g7lj\" (UniqueName: \"kubernetes.io/projected/f59e0fc2-4364-4718-9c16-34bf2c019278-kube-api-access-2g7lj\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:44.626250 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:44.626228 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls podName:f59e0fc2-4364-4718-9c16-34bf2c019278 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:45.126207982 +0000 UTC m=+35.143437319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls") pod "dns-default-85spg" (UID: "f59e0fc2-4364-4718-9c16-34bf2c019278") : secret "dns-default-metrics-tls" not found Jul 16 17:34:44.626391 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.626299 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjnh9\" (UniqueName: \"kubernetes.io/projected/142b45fe-5371-4dc2-998b-a4aedafa976c-kube-api-access-zjnh9\") pod \"ingress-canary-5v4vt\" (UID: \"142b45fe-5371-4dc2-998b-a4aedafa976c\") " pod="openshift-ingress-canary/ingress-canary-5v4vt" Jul 16 17:34:44.626391 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.626321 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59e0fc2-4364-4718-9c16-34bf2c019278-tmp-dir\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:44.626614 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.626595 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f59e0fc2-4364-4718-9c16-34bf2c019278-config-volume\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:44.630277 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.630258 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/142b45fe-5371-4dc2-998b-a4aedafa976c-cert\") pod \"ingress-canary-5v4vt\" (UID: \"142b45fe-5371-4dc2-998b-a4aedafa976c\") " pod="openshift-ingress-canary/ingress-canary-5v4vt" Jul 16 17:34:44.635006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.634988 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjnh9\" (UniqueName: \"kubernetes.io/projected/142b45fe-5371-4dc2-998b-a4aedafa976c-kube-api-access-zjnh9\") pod \"ingress-canary-5v4vt\" (UID: \"142b45fe-5371-4dc2-998b-a4aedafa976c\") " pod="openshift-ingress-canary/ingress-canary-5v4vt" Jul 16 17:34:44.635228 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.635208 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g7lj\" (UniqueName: \"kubernetes.io/projected/f59e0fc2-4364-4718-9c16-34bf2c019278-kube-api-access-2g7lj\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:44.695224 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.695172 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5v4vt" Jul 16 17:34:44.731082 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.731059 2582 generic.go:356] "Generic (PLEG): container finished" podID="63aefcda-40ba-4e5d-8fd4-491a9bf5b45b" containerID="e6521d3c55e658cb971477bf006981f544fc528190ae4044ba6c10b4f8028b7e" exitCode=0 Jul 16 17:34:44.731165 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.731116 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct6j5" event={"ID":"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b","Type":"ContainerDied","Data":"e6521d3c55e658cb971477bf006981f544fc528190ae4044ba6c10b4f8028b7e"} Jul 16 17:34:44.875007 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:44.874851 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5v4vt"] Jul 16 17:34:44.878243 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:34:44.878210 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142b45fe_5371_4dc2_998b_a4aedafa976c.slice/crio-a6d87d32a1adc450c6dbc6348694fa96fa847312ead6c6f21ea2b67d338fe463 WatchSource:0}: Error finding container a6d87d32a1adc450c6dbc6348694fa96fa847312ead6c6f21ea2b67d338fe463: Status 404 returned error can't find the container with id a6d87d32a1adc450c6dbc6348694fa96fa847312ead6c6f21ea2b67d338fe463 Jul 16 17:34:45.129909 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:45.129881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:45.130496 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:45.130041 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:34:45.130496 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:45.130115 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls podName:f59e0fc2-4364-4718-9c16-34bf2c019278 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:46.130092728 +0000 UTC m=+36.147322050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls") pod "dns-default-85spg" (UID: "f59e0fc2-4364-4718-9c16-34bf2c019278") : secret "dns-default-metrics-tls" not found Jul 16 17:34:45.734654 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:45.734618 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5v4vt" event={"ID":"142b45fe-5371-4dc2-998b-a4aedafa976c","Type":"ContainerStarted","Data":"a6d87d32a1adc450c6dbc6348694fa96fa847312ead6c6f21ea2b67d338fe463"} Jul 16 17:34:46.137381 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:46.137296 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:46.137881 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:46.137484 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:34:46.137881 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:46.137562 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls podName:f59e0fc2-4364-4718-9c16-34bf2c019278 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:48.137541141 +0000 UTC m=+38.154770488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls") pod "dns-default-85spg" (UID: "f59e0fc2-4364-4718-9c16-34bf2c019278") : secret "dns-default-metrics-tls" not found Jul 16 17:34:47.742376 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:47.741898 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5v4vt" event={"ID":"142b45fe-5371-4dc2-998b-a4aedafa976c","Type":"ContainerStarted","Data":"3c779f7b38c8eb73f9ed22e12b5d755dffefa3ef81e6b33f5d2a289ab4cb6810"} Jul 16 17:34:47.761217 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:47.761164 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5v4vt" podStartSLOduration=1.171233266 podStartE2EDuration="3.76114331s" totalImagesPullingTime="2.589910044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:47.761138505 +0000 UTC m=+37.778367849" watchObservedRunningTime="2026-07-16 17:34:47.76114331 +0000 UTC m=+37.778372654" Jul 16 17:34:48.150871 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:48.150837 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:48.151019 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:48.150953 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:34:48.151019 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:48.151013 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls podName:f59e0fc2-4364-4718-9c16-34bf2c019278 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:52.150997868 +0000 UTC m=+42.168227189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls") pod "dns-default-85spg" (UID: "f59e0fc2-4364-4718-9c16-34bf2c019278") : secret "dns-default-metrics-tls" not found Jul 16 17:34:51.753902 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:51.753868 2582 generic.go:356] "Generic (PLEG): container finished" podID="63aefcda-40ba-4e5d-8fd4-491a9bf5b45b" containerID="7997ff057f3a2611157042f6ad9963259b65b2df447b2e579ba710018d1b14e2" exitCode=0 Jul 16 17:34:51.754225 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:51.753931 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct6j5" event={"ID":"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b","Type":"ContainerDied","Data":"7997ff057f3a2611157042f6ad9963259b65b2df447b2e579ba710018d1b14e2"} Jul 16 17:34:52.181588 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:52.181520 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:34:52.181697 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:52.181662 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:34:52.181735 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:34:52.181726 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls podName:f59e0fc2-4364-4718-9c16-34bf2c019278 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:00.181708322 +0000 UTC m=+50.198937665 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls") pod "dns-default-85spg" (UID: "f59e0fc2-4364-4718-9c16-34bf2c019278") : secret "dns-default-metrics-tls" not found Jul 16 17:34:52.759870 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:52.759827 2582 generic.go:356] "Generic (PLEG): container finished" podID="63aefcda-40ba-4e5d-8fd4-491a9bf5b45b" containerID="9346bbe8f798b5e58fdf0df3e394c6df851d62b276e28e6ba3a2b6e1e2585563" exitCode=0 Jul 16 17:34:52.760350 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:52.759875 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct6j5" event={"ID":"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b","Type":"ContainerDied","Data":"9346bbe8f798b5e58fdf0df3e394c6df851d62b276e28e6ba3a2b6e1e2585563"} Jul 16 17:34:53.765151 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:53.765118 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct6j5" event={"ID":"63aefcda-40ba-4e5d-8fd4-491a9bf5b45b","Type":"ContainerStarted","Data":"92769a7052941ee6360110366f10f6999e1f8639a1ea73ec619f328b92899c1c"} Jul 16 17:34:53.790690 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:34:53.790648 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ct6j5" podStartSLOduration=9.221565423 podStartE2EDuration="43.790634688s" totalImagesPullingTime="34.569069265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:53.790407316 +0000 UTC m=+43.807636657" watchObservedRunningTime="2026-07-16 17:34:53.790634688 +0000 UTC m=+43.807864029" Jul 16 17:35:00.227827 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:00.227776 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:35:00.228387 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:00.227995 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:35:00.228387 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:00.228090 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls podName:f59e0fc2-4364-4718-9c16-34bf2c019278 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:16.228066903 +0000 UTC m=+66.245296223 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls") pod "dns-default-85spg" (UID: "f59e0fc2-4364-4718-9c16-34bf2c019278") : secret "dns-default-metrics-tls" not found Jul 16 17:35:08.723898 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:08.723860 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4lfqt" Jul 16 17:35:15.444728 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.444684 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tmzwf"] Jul 16 17:35:15.448446 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.448415 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tmzwf" Jul 16 17:35:15.451142 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.451105 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-cr7dt\"" Jul 16 17:35:15.451327 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.451277 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:35:15.452095 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.452078 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 16 17:35:15.456948 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.456919 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tmzwf"] Jul 16 17:35:15.530455 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.530433 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxxmb\" (UniqueName: \"kubernetes.io/projected/a724e2d9-c8a0-4cbf-b79b-3463893c6019-kube-api-access-hxxmb\") pod \"volume-data-source-validator-78495597f6-tmzwf\" (UID: \"a724e2d9-c8a0-4cbf-b79b-3463893c6019\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tmzwf" Jul 16 17:35:15.550404 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.550368 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-2wxlk"] Jul 16 17:35:15.552132 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.552114 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d"] Jul 16 17:35:15.552271 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.552251 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.553949 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.553929 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:15.557602 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.557583 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:35:15.558039 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.558023 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 17:35:15.558264 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.558252 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cps67\"" Jul 16 17:35:15.558409 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.558398 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pp7zn\"" Jul 16 17:35:15.558829 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.558811 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 16 17:35:15.558932 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.558914 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 16 17:35:15.559017 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.558995 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 17:35:15.559175 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.559089 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 16 17:35:15.559175 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.559124 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 16 17:35:15.561262 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.561245 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 16 17:35:15.564948 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.564929 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-2wxlk"] Jul 16 17:35:15.570909 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.570892 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 16 17:35:15.581204 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.581184 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d"] Jul 16 17:35:15.630922 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.630899 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c223b307-534a-47e4-b677-e015ec5e6861-config\") pod \"console-operator-7866497494-2wxlk\" (UID: \"c223b307-534a-47e4-b677-e015ec5e6861\") " pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.631000 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.630926 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c223b307-534a-47e4-b677-e015ec5e6861-trusted-ca\") pod \"console-operator-7866497494-2wxlk\" (UID: \"c223b307-534a-47e4-b677-e015ec5e6861\") " pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.631000 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.630951 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c223b307-534a-47e4-b677-e015ec5e6861-serving-cert\") pod \"console-operator-7866497494-2wxlk\" (UID: \"c223b307-534a-47e4-b677-e015ec5e6861\") " pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.631000 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.630967 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdmj4\" (UniqueName: \"kubernetes.io/projected/c223b307-534a-47e4-b677-e015ec5e6861-kube-api-access-hdmj4\") pod \"console-operator-7866497494-2wxlk\" (UID: \"c223b307-534a-47e4-b677-e015ec5e6861\") " pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.631105 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.631030 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxxmb\" (UniqueName: \"kubernetes.io/projected/a724e2d9-c8a0-4cbf-b79b-3463893c6019-kube-api-access-hxxmb\") pod \"volume-data-source-validator-78495597f6-tmzwf\" (UID: \"a724e2d9-c8a0-4cbf-b79b-3463893c6019\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tmzwf" Jul 16 17:35:15.643565 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.643545 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxxmb\" (UniqueName: \"kubernetes.io/projected/a724e2d9-c8a0-4cbf-b79b-3463893c6019-kube-api-access-hxxmb\") pod \"volume-data-source-validator-78495597f6-tmzwf\" (UID: \"a724e2d9-c8a0-4cbf-b79b-3463893c6019\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tmzwf" Jul 16 17:35:15.647280 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.647258 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf"] Jul 16 17:35:15.649339 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.649324 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" Jul 16 17:35:15.651900 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.651880 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 16 17:35:15.651997 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.651910 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 16 17:35:15.651997 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.651934 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-c8tsz\"" Jul 16 17:35:15.652201 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.652186 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:35:15.652273 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.652219 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 16 17:35:15.661415 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.661394 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf"] Jul 16 17:35:15.732176 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.732154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ec09f0e-e2cc-4e21-9200-1b91486a5251-serving-cert\") pod \"service-ca-operator-7695fc6c8c-rf6rf\" (UID: \"2ec09f0e-e2cc-4e21-9200-1b91486a5251\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" Jul 16 17:35:15.732262 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.732181 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ec09f0e-e2cc-4e21-9200-1b91486a5251-config\") pod \"service-ca-operator-7695fc6c8c-rf6rf\" (UID: \"2ec09f0e-e2cc-4e21-9200-1b91486a5251\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" Jul 16 17:35:15.732262 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.732201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c223b307-534a-47e4-b677-e015ec5e6861-config\") pod \"console-operator-7866497494-2wxlk\" (UID: \"c223b307-534a-47e4-b677-e015ec5e6861\") " pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.732262 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.732216 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:15.732361 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.732276 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c223b307-534a-47e4-b677-e015ec5e6861-trusted-ca\") pod \"console-operator-7866497494-2wxlk\" (UID: \"c223b307-534a-47e4-b677-e015ec5e6861\") " pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.732398 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.732363 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c223b307-534a-47e4-b677-e015ec5e6861-serving-cert\") pod \"console-operator-7866497494-2wxlk\" (UID: \"c223b307-534a-47e4-b677-e015ec5e6861\") " pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.732440 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.732402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmj4\" (UniqueName: \"kubernetes.io/projected/c223b307-534a-47e4-b677-e015ec5e6861-kube-api-access-hdmj4\") pod \"console-operator-7866497494-2wxlk\" (UID: \"c223b307-534a-47e4-b677-e015ec5e6861\") " pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.732440 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.732432 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsxpf\" (UniqueName: \"kubernetes.io/projected/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-kube-api-access-dsxpf\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:15.732558 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.732466 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcb4j\" (UniqueName: \"kubernetes.io/projected/2ec09f0e-e2cc-4e21-9200-1b91486a5251-kube-api-access-vcb4j\") pod \"service-ca-operator-7695fc6c8c-rf6rf\" (UID: \"2ec09f0e-e2cc-4e21-9200-1b91486a5251\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" Jul 16 17:35:15.732616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.732577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:15.732812 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.732786 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c223b307-534a-47e4-b677-e015ec5e6861-config\") pod \"console-operator-7866497494-2wxlk\" (UID: \"c223b307-534a-47e4-b677-e015ec5e6861\") " pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.733140 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.733122 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c223b307-534a-47e4-b677-e015ec5e6861-trusted-ca\") pod \"console-operator-7866497494-2wxlk\" (UID: \"c223b307-534a-47e4-b677-e015ec5e6861\") " pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.734701 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.734685 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c223b307-534a-47e4-b677-e015ec5e6861-serving-cert\") pod \"console-operator-7866497494-2wxlk\" (UID: \"c223b307-534a-47e4-b677-e015ec5e6861\") " pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.741830 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.741811 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdmj4\" (UniqueName: \"kubernetes.io/projected/c223b307-534a-47e4-b677-e015ec5e6861-kube-api-access-hdmj4\") pod \"console-operator-7866497494-2wxlk\" (UID: \"c223b307-534a-47e4-b677-e015ec5e6861\") " pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.758734 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.758715 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tmzwf" Jul 16 17:35:15.832950 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.832919 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:15.833053 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.832997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ec09f0e-e2cc-4e21-9200-1b91486a5251-serving-cert\") pod \"service-ca-operator-7695fc6c8c-rf6rf\" (UID: \"2ec09f0e-e2cc-4e21-9200-1b91486a5251\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" Jul 16 17:35:15.833053 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.833025 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ec09f0e-e2cc-4e21-9200-1b91486a5251-config\") pod \"service-ca-operator-7695fc6c8c-rf6rf\" (UID: \"2ec09f0e-e2cc-4e21-9200-1b91486a5251\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" Jul 16 17:35:15.833119 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.833053 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:15.833119 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.833101 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsxpf\" (UniqueName: \"kubernetes.io/projected/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-kube-api-access-dsxpf\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:15.833693 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:15.833219 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 17:35:15.833693 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:15.833295 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls podName:b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:16.333275646 +0000 UTC m=+66.350504969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-bgl8d" (UID: "b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0") : secret "cluster-monitoring-operator-tls" not found Jul 16 17:35:15.833693 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.833616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcb4j\" (UniqueName: \"kubernetes.io/projected/2ec09f0e-e2cc-4e21-9200-1b91486a5251-kube-api-access-vcb4j\") pod \"service-ca-operator-7695fc6c8c-rf6rf\" (UID: \"2ec09f0e-e2cc-4e21-9200-1b91486a5251\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" Jul 16 17:35:15.833912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.833779 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ec09f0e-e2cc-4e21-9200-1b91486a5251-config\") pod \"service-ca-operator-7695fc6c8c-rf6rf\" (UID: \"2ec09f0e-e2cc-4e21-9200-1b91486a5251\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" Jul 16 17:35:15.833961 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.833918 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:15.835847 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.835812 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ec09f0e-e2cc-4e21-9200-1b91486a5251-serving-cert\") pod \"service-ca-operator-7695fc6c8c-rf6rf\" (UID: \"2ec09f0e-e2cc-4e21-9200-1b91486a5251\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" Jul 16 17:35:15.846327 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.846281 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcb4j\" (UniqueName: \"kubernetes.io/projected/2ec09f0e-e2cc-4e21-9200-1b91486a5251-kube-api-access-vcb4j\") pod \"service-ca-operator-7695fc6c8c-rf6rf\" (UID: \"2ec09f0e-e2cc-4e21-9200-1b91486a5251\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" Jul 16 17:35:15.846327 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.846293 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsxpf\" (UniqueName: \"kubernetes.io/projected/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-kube-api-access-dsxpf\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:15.862720 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.862489 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:15.885014 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.884993 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tmzwf"] Jul 16 17:35:15.887967 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:35:15.887945 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda724e2d9_c8a0_4cbf_b79b_3463893c6019.slice/crio-e8767563525ca7306e5000981815066af2862ec2659909db4d1b4543b6a3889e WatchSource:0}: Error finding container e8767563525ca7306e5000981815066af2862ec2659909db4d1b4543b6a3889e: Status 404 returned error can't find the container with id e8767563525ca7306e5000981815066af2862ec2659909db4d1b4543b6a3889e Jul 16 17:35:15.957912 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.957882 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" Jul 16 17:35:15.991769 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:15.991690 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-2wxlk"] Jul 16 17:35:15.998388 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:35:15.998362 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc223b307_534a_47e4_b677_e015ec5e6861.slice/crio-fb1796c95b9efbe8d1480912d26363e659d899f6f45357244e34eda831e83ba9 WatchSource:0}: Error finding container fb1796c95b9efbe8d1480912d26363e659d899f6f45357244e34eda831e83ba9: Status 404 returned error can't find the container with id fb1796c95b9efbe8d1480912d26363e659d899f6f45357244e34eda831e83ba9 Jul 16 17:35:16.077832 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.077805 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf"] Jul 16 17:35:16.080980 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:35:16.080956 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec09f0e_e2cc_4e21_9200_1b91486a5251.slice/crio-2d5663199d5fb23a8bb0795283bbdf3da900e8cae5759df4ac23ca743e4fb0a3 WatchSource:0}: Error finding container 2d5663199d5fb23a8bb0795283bbdf3da900e8cae5759df4ac23ca743e4fb0a3: Status 404 returned error can't find the container with id 2d5663199d5fb23a8bb0795283bbdf3da900e8cae5759df4ac23ca743e4fb0a3 Jul 16 17:35:16.136741 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.136717 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:35:16.136826 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.136764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:35:16.139423 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.139397 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 17:35:16.139423 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.139414 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 17:35:16.147271 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:16.147253 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 17:35:16.147352 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:16.147304 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs podName:dc349e33-d51c-40bf-8890-392c3393299d nodeName:}" failed. No retries permitted until 2026-07-16 17:36:20.147289245 +0000 UTC m=+130.164518569 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs") pod "network-metrics-daemon-5qrcw" (UID: "dc349e33-d51c-40bf-8890-392c3393299d") : secret "metrics-daemon-secret" not found Jul 16 17:35:16.149426 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.149409 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503d02a0-3e8a-4786-bf7e-c4c0bec8dbda-original-pull-secret\") pod \"global-pull-secret-syncer-hfrlv\" (UID: \"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda\") " pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:35:16.237358 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.237330 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:35:16.237494 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:16.237460 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:35:16.237588 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:16.237530 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls podName:f59e0fc2-4364-4718-9c16-34bf2c019278 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:48.237516008 +0000 UTC m=+98.254745328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls") pod "dns-default-85spg" (UID: "f59e0fc2-4364-4718-9c16-34bf2c019278") : secret "dns-default-metrics-tls" not found Jul 16 17:35:16.338025 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.337958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:16.338025 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.337998 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22dw\" (UniqueName: \"kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw\") pod \"network-check-target-x2vn2\" (UID: \"458ea307-1c97-4cda-8723-97e047f76a50\") " pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:35:16.338149 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:16.338118 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 17:35:16.338190 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:16.338177 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls podName:b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:17.338160677 +0000 UTC m=+67.355389997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-bgl8d" (UID: "b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0") : secret "cluster-monitoring-operator-tls" not found Jul 16 17:35:16.340767 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.340748 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 17:35:16.350896 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.350876 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 17:35:16.361467 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.361446 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m22dw\" (UniqueName: \"kubernetes.io/projected/458ea307-1c97-4cda-8723-97e047f76a50-kube-api-access-m22dw\") pod \"network-check-target-x2vn2\" (UID: \"458ea307-1c97-4cda-8723-97e047f76a50\") " pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:35:16.389199 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.389172 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfrlv" Jul 16 17:35:16.398981 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.398010 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s9dx5\"" Jul 16 17:35:16.406425 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.406406 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:35:16.554325 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.554297 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hfrlv"] Jul 16 17:35:16.557669 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:35:16.557643 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503d02a0_3e8a_4786_bf7e_c4c0bec8dbda.slice/crio-4d9b85812cc4dc6182ab23d20ce49a21a8775f9041347a38d623d042d47083f0 WatchSource:0}: Error finding container 4d9b85812cc4dc6182ab23d20ce49a21a8775f9041347a38d623d042d47083f0: Status 404 returned error can't find the container with id 4d9b85812cc4dc6182ab23d20ce49a21a8775f9041347a38d623d042d47083f0 Jul 16 17:35:16.571025 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.571001 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x2vn2"] Jul 16 17:35:16.574352 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:35:16.574323 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458ea307_1c97_4cda_8723_97e047f76a50.slice/crio-36343b9cb229ec36693d1618dc98c323119e68469f270c258defc1c5625e3e45 WatchSource:0}: Error finding container 36343b9cb229ec36693d1618dc98c323119e68469f270c258defc1c5625e3e45: Status 404 returned error can't find the container with id 36343b9cb229ec36693d1618dc98c323119e68469f270c258defc1c5625e3e45 Jul 16 17:35:16.821093 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.821049 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x2vn2" event={"ID":"458ea307-1c97-4cda-8723-97e047f76a50","Type":"ContainerStarted","Data":"36343b9cb229ec36693d1618dc98c323119e68469f270c258defc1c5625e3e45"} Jul 16 17:35:16.822489 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.822449 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hfrlv" event={"ID":"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda","Type":"ContainerStarted","Data":"4d9b85812cc4dc6182ab23d20ce49a21a8775f9041347a38d623d042d47083f0"} Jul 16 17:35:16.823858 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.823828 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-2wxlk" event={"ID":"c223b307-534a-47e4-b677-e015ec5e6861","Type":"ContainerStarted","Data":"fb1796c95b9efbe8d1480912d26363e659d899f6f45357244e34eda831e83ba9"} Jul 16 17:35:16.825589 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.825561 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tmzwf" event={"ID":"a724e2d9-c8a0-4cbf-b79b-3463893c6019","Type":"ContainerStarted","Data":"e8767563525ca7306e5000981815066af2862ec2659909db4d1b4543b6a3889e"} Jul 16 17:35:16.827095 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:16.827070 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" event={"ID":"2ec09f0e-e2cc-4e21-9200-1b91486a5251","Type":"ContainerStarted","Data":"2d5663199d5fb23a8bb0795283bbdf3da900e8cae5759df4ac23ca743e4fb0a3"} Jul 16 17:35:17.347651 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:17.347564 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:17.347811 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:17.347749 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 17:35:17.347879 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:17.347811 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls podName:b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:19.347793305 +0000 UTC m=+69.365022627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-bgl8d" (UID: "b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0") : secret "cluster-monitoring-operator-tls" not found Jul 16 17:35:17.832384 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:17.832315 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tmzwf" event={"ID":"a724e2d9-c8a0-4cbf-b79b-3463893c6019","Type":"ContainerStarted","Data":"398cbd4231a0908a917bf79cc1927b175d0f9a19d1d4aa468a5dcd83a6a1f12f"} Jul 16 17:35:17.853746 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:17.853517 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tmzwf" podStartSLOduration=1.7357858739999998 podStartE2EDuration="2.853495154s" totalImagesPullingTime="1.11770928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:17.852292605 +0000 UTC m=+67.869521948" watchObservedRunningTime="2026-07-16 17:35:17.853495154 +0000 UTC m=+67.870724500" Jul 16 17:35:19.013088 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.013045 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hsd9q"] Jul 16 17:35:19.016003 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.015974 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.018891 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.018867 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 17:35:19.018891 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.018891 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 17:35:19.019192 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.019173 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 17:35:19.019304 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.019182 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 17:35:19.020181 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.020004 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lbz87\"" Jul 16 17:35:19.027998 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.027960 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hsd9q"] Jul 16 17:35:19.163502 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.163221 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.163502 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.163272 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb64s\" (UniqueName: \"kubernetes.io/projected/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-api-access-qb64s\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.163502 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.163306 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.163502 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.163352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c05683ac-5e47-45a9-8de4-90bcb06edc11-crio-socket\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.163502 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.163368 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c05683ac-5e47-45a9-8de4-90bcb06edc11-data-volume\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.264447 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.264372 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.264447 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.264418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb64s\" (UniqueName: \"kubernetes.io/projected/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-api-access-qb64s\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.264647 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.264451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.264647 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.264532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c05683ac-5e47-45a9-8de4-90bcb06edc11-crio-socket\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.264647 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:19.264553 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:35:19.264647 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:19.264630 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls podName:c05683ac-5e47-45a9-8de4-90bcb06edc11 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:19.764609778 +0000 UTC m=+69.781839117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hsd9q" (UID: "c05683ac-5e47-45a9-8de4-90bcb06edc11") : secret "insights-runtime-extractor-tls" not found Jul 16 17:35:19.264647 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.264556 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c05683ac-5e47-45a9-8de4-90bcb06edc11-data-volume\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.264878 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.264854 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c05683ac-5e47-45a9-8de4-90bcb06edc11-data-volume\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.265149 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:19.264996 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:35:19.265149 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:19.265055 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm podName:c05683ac-5e47-45a9-8de4-90bcb06edc11 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:19.765037992 +0000 UTC m=+69.782267334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hsd9q" (UID: "c05683ac-5e47-45a9-8de4-90bcb06edc11") : configmap "kube-rbac-proxy" not found Jul 16 17:35:19.265149 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.265120 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c05683ac-5e47-45a9-8de4-90bcb06edc11-crio-socket\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.279955 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.279932 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb64s\" (UniqueName: \"kubernetes.io/projected/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-api-access-qb64s\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.308369 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.308334 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-k4gc5"] Jul 16 17:35:19.311778 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.311756 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:19.314142 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.314121 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 16 17:35:19.314449 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.314430 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 16 17:35:19.314641 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.314623 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xrtcx\"" Jul 16 17:35:19.323076 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.323056 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-k4gc5"] Jul 16 17:35:19.365568 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.365521 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:19.365737 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:19.365720 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 17:35:19.365821 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:19.365780 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls podName:b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:23.365767183 +0000 UTC m=+73.382996506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-bgl8d" (UID: "b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0") : secret "cluster-monitoring-operator-tls" not found Jul 16 17:35:19.406692 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.406661 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kv4hq"] Jul 16 17:35:19.408937 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.408915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kv4hq" Jul 16 17:35:19.411543 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.411523 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-x6g22\"" Jul 16 17:35:19.411642 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.411547 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 16 17:35:19.412086 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.412071 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 16 17:35:19.418635 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.418612 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kv4hq"] Jul 16 17:35:19.466599 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.466569 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/edd90096-a7e6-4040-a60c-76b5030fe275-nginx-conf\") pod \"networking-console-plugin-6cf856758c-k4gc5\" (UID: \"edd90096-a7e6-4040-a60c-76b5030fe275\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:19.466762 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.466712 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k4gc5\" (UID: \"edd90096-a7e6-4040-a60c-76b5030fe275\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:19.567900 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.567809 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k4gc5\" (UID: \"edd90096-a7e6-4040-a60c-76b5030fe275\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:19.567900 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.567861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zpp\" (UniqueName: \"kubernetes.io/projected/63681960-82a8-45eb-b340-adfa536295c3-kube-api-access-z7zpp\") pod \"migrator-7f59b54f6c-kv4hq\" (UID: \"63681960-82a8-45eb-b340-adfa536295c3\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kv4hq" Jul 16 17:35:19.568068 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.567936 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/edd90096-a7e6-4040-a60c-76b5030fe275-nginx-conf\") pod \"networking-console-plugin-6cf856758c-k4gc5\" (UID: \"edd90096-a7e6-4040-a60c-76b5030fe275\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:19.568068 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:19.567993 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 17:35:19.568068 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:19.568054 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert podName:edd90096-a7e6-4040-a60c-76b5030fe275 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:20.0680375 +0000 UTC m=+70.085266820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-k4gc5" (UID: "edd90096-a7e6-4040-a60c-76b5030fe275") : secret "networking-console-plugin-cert" not found Jul 16 17:35:19.568581 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.568557 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/edd90096-a7e6-4040-a60c-76b5030fe275-nginx-conf\") pod \"networking-console-plugin-6cf856758c-k4gc5\" (UID: \"edd90096-a7e6-4040-a60c-76b5030fe275\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:19.668413 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.668378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zpp\" (UniqueName: \"kubernetes.io/projected/63681960-82a8-45eb-b340-adfa536295c3-kube-api-access-z7zpp\") pod \"migrator-7f59b54f6c-kv4hq\" (UID: \"63681960-82a8-45eb-b340-adfa536295c3\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kv4hq" Jul 16 17:35:19.678426 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.678388 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zpp\" (UniqueName: \"kubernetes.io/projected/63681960-82a8-45eb-b340-adfa536295c3-kube-api-access-z7zpp\") pod \"migrator-7f59b54f6c-kv4hq\" (UID: \"63681960-82a8-45eb-b340-adfa536295c3\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kv4hq" Jul 16 17:35:19.720980 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.720954 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kv4hq" Jul 16 17:35:19.769043 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.769012 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.769157 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:19.769061 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:19.769217 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:19.769158 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:35:19.769217 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:19.769162 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:35:19.769217 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:19.769211 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm podName:c05683ac-5e47-45a9-8de4-90bcb06edc11 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:20.769195975 +0000 UTC m=+70.786425301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hsd9q" (UID: "c05683ac-5e47-45a9-8de4-90bcb06edc11") : configmap "kube-rbac-proxy" not found Jul 16 17:35:19.769458 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:19.769225 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls podName:c05683ac-5e47-45a9-8de4-90bcb06edc11 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:20.769218997 +0000 UTC m=+70.786448317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hsd9q" (UID: "c05683ac-5e47-45a9-8de4-90bcb06edc11") : secret "insights-runtime-extractor-tls" not found Jul 16 17:35:20.071282 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:20.071236 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k4gc5\" (UID: \"edd90096-a7e6-4040-a60c-76b5030fe275\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:20.071744 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:20.071424 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 17:35:20.071744 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:20.071526 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert podName:edd90096-a7e6-4040-a60c-76b5030fe275 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:21.071504914 +0000 UTC m=+71.088734234 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-k4gc5" (UID: "edd90096-a7e6-4040-a60c-76b5030fe275") : secret "networking-console-plugin-cert" not found Jul 16 17:35:20.779304 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:20.779265 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:20.779497 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:20.779318 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:20.779497 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:20.779397 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:35:20.779497 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:20.779428 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:35:20.779497 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:20.779486 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm podName:c05683ac-5e47-45a9-8de4-90bcb06edc11 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:22.779450084 +0000 UTC m=+72.796679426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hsd9q" (UID: "c05683ac-5e47-45a9-8de4-90bcb06edc11") : configmap "kube-rbac-proxy" not found Jul 16 17:35:20.779662 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:20.779509 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls podName:c05683ac-5e47-45a9-8de4-90bcb06edc11 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:22.779498049 +0000 UTC m=+72.796727370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hsd9q" (UID: "c05683ac-5e47-45a9-8de4-90bcb06edc11") : secret "insights-runtime-extractor-tls" not found Jul 16 17:35:20.989246 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:20.989204 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d7b2n_782eb993-d0db-4c25-b990-34dbe4d7a5b9/dns-node-resolver/0.log" Jul 16 17:35:21.082732 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.082649 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k4gc5\" (UID: \"edd90096-a7e6-4040-a60c-76b5030fe275\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:21.083148 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:21.082805 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 17:35:21.083148 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:21.082876 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert podName:edd90096-a7e6-4040-a60c-76b5030fe275 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:23.0828574 +0000 UTC m=+73.100086725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-k4gc5" (UID: "edd90096-a7e6-4040-a60c-76b5030fe275") : secret "networking-console-plugin-cert" not found Jul 16 17:35:21.659097 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.659012 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kv4hq"] Jul 16 17:35:21.666051 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:35:21.666023 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63681960_82a8_45eb_b340_adfa536295c3.slice/crio-14b3dd166fea3299e45dd14d07ee09c76811515780b3f2dee84d3fa0c700859f WatchSource:0}: Error finding container 14b3dd166fea3299e45dd14d07ee09c76811515780b3f2dee84d3fa0c700859f: Status 404 returned error can't find the container with id 14b3dd166fea3299e45dd14d07ee09c76811515780b3f2dee84d3fa0c700859f Jul 16 17:35:21.789289 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.789227 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4ptfz_5fa87729-4cd3-4453-a444-9c59a6514753/node-ca/0.log" Jul 16 17:35:21.846654 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.846619 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kv4hq" event={"ID":"63681960-82a8-45eb-b340-adfa536295c3","Type":"ContainerStarted","Data":"14b3dd166fea3299e45dd14d07ee09c76811515780b3f2dee84d3fa0c700859f"} Jul 16 17:35:21.848258 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.848232 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" event={"ID":"2ec09f0e-e2cc-4e21-9200-1b91486a5251","Type":"ContainerStarted","Data":"ad99ad66b034d1c0ce2ab81c662a35a2811b1b96e674004e9eeb189994bcc5ca"} Jul 16 17:35:21.849826 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.849802 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x2vn2" event={"ID":"458ea307-1c97-4cda-8723-97e047f76a50","Type":"ContainerStarted","Data":"2a51383da0e29e3dad8ffd486f9ee0244f05a046009597f96f9a713f9712f62f"} Jul 16 17:35:21.849923 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.849855 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:35:21.851346 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.851318 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hfrlv" event={"ID":"503d02a0-3e8a-4786-bf7e-c4c0bec8dbda","Type":"ContainerStarted","Data":"437c9faf64f3be5f7e3ac2c0af704409e3e06245633bcce83d180c712f6d99e8"} Jul 16 17:35:21.852913 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.852893 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/0.log" Jul 16 17:35:21.852975 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.852932 2582 generic.go:356] "Generic (PLEG): container finished" podID="c223b307-534a-47e4-b677-e015ec5e6861" containerID="d1ed8b304bf07fe3a164cf7225245016cd5c74a211c1b28bbc3482965a12d8a9" exitCode=255 Jul 16 17:35:21.852975 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.852963 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-2wxlk" event={"ID":"c223b307-534a-47e4-b677-e015ec5e6861","Type":"ContainerDied","Data":"d1ed8b304bf07fe3a164cf7225245016cd5c74a211c1b28bbc3482965a12d8a9"} Jul 16 17:35:21.853205 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.853190 2582 scope.go:117] "RemoveContainer" containerID="d1ed8b304bf07fe3a164cf7225245016cd5c74a211c1b28bbc3482965a12d8a9" Jul 16 17:35:21.871579 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.871536 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" podStartSLOduration=1.433854923 podStartE2EDuration="6.871519388s" totalImagesPullingTime="5.437664465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:21.870314137 +0000 UTC m=+71.887543470" watchObservedRunningTime="2026-07-16 17:35:21.871519388 +0000 UTC m=+71.888748731" Jul 16 17:35:21.887358 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.887223 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hfrlv" podStartSLOduration=66.842456983 podStartE2EDuration="1m11.887212194s" totalImagesPullingTime="5.044755211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:21.886727986 +0000 UTC m=+71.903957327" watchObservedRunningTime="2026-07-16 17:35:21.887212194 +0000 UTC m=+71.904441535" Jul 16 17:35:21.903768 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:21.903732 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x2vn2" podStartSLOduration=66.95432818 podStartE2EDuration="1m11.903719404s" totalImagesPullingTime="4.949391224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:21.903335632 +0000 UTC m=+71.920564995" watchObservedRunningTime="2026-07-16 17:35:21.903719404 +0000 UTC m=+71.920948723" Jul 16 17:35:22.796014 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:22.795934 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:22.796014 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:22.795977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:22.796497 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:22.796078 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:35:22.796497 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:22.796138 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm podName:c05683ac-5e47-45a9-8de4-90bcb06edc11 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:26.796125271 +0000 UTC m=+76.813354590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hsd9q" (UID: "c05683ac-5e47-45a9-8de4-90bcb06edc11") : configmap "kube-rbac-proxy" not found Jul 16 17:35:22.796497 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:22.796146 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:35:22.796497 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:22.796232 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls podName:c05683ac-5e47-45a9-8de4-90bcb06edc11 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:26.796211182 +0000 UTC m=+76.813440518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hsd9q" (UID: "c05683ac-5e47-45a9-8de4-90bcb06edc11") : secret "insights-runtime-extractor-tls" not found Jul 16 17:35:22.858134 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:22.858110 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 17:35:22.858503 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:22.858485 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/0.log" Jul 16 17:35:22.858608 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:22.858526 2582 generic.go:356] "Generic (PLEG): container finished" podID="c223b307-534a-47e4-b677-e015ec5e6861" containerID="d52e8afc420880b0e4e36da142e2924994e21419de33b0a04a6b22a3fd6db7fc" exitCode=255 Jul 16 17:35:22.858655 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:22.858622 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-2wxlk" event={"ID":"c223b307-534a-47e4-b677-e015ec5e6861","Type":"ContainerDied","Data":"d52e8afc420880b0e4e36da142e2924994e21419de33b0a04a6b22a3fd6db7fc"} Jul 16 17:35:22.858704 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:22.858690 2582 scope.go:117] "RemoveContainer" containerID="d1ed8b304bf07fe3a164cf7225245016cd5c74a211c1b28bbc3482965a12d8a9" Jul 16 17:35:22.858847 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:22.858823 2582 scope.go:117] "RemoveContainer" containerID="d52e8afc420880b0e4e36da142e2924994e21419de33b0a04a6b22a3fd6db7fc" Jul 16 17:35:22.859025 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:22.859001 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-2wxlk_openshift-console-operator(c223b307-534a-47e4-b677-e015ec5e6861)\"" pod="openshift-console-operator/console-operator-7866497494-2wxlk" podUID="c223b307-534a-47e4-b677-e015ec5e6861" Jul 16 17:35:22.988509 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:22.988465 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5v4vt_142b45fe-5371-4dc2-998b-a4aedafa976c/serve-healthcheck-canary/0.log" Jul 16 17:35:23.098895 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:23.098846 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k4gc5\" (UID: \"edd90096-a7e6-4040-a60c-76b5030fe275\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:23.098961 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:23.098945 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 17:35:23.099000 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:23.098987 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert podName:edd90096-a7e6-4040-a60c-76b5030fe275 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:27.09897634 +0000 UTC m=+77.116205659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-k4gc5" (UID: "edd90096-a7e6-4040-a60c-76b5030fe275") : secret "networking-console-plugin-cert" not found Jul 16 17:35:23.401459 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:23.401383 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:23.401614 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:23.401574 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 17:35:23.401669 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:23.401652 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls podName:b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:31.401632102 +0000 UTC m=+81.418861444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-bgl8d" (UID: "b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0") : secret "cluster-monitoring-operator-tls" not found Jul 16 17:35:23.863956 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:23.863927 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 17:35:23.864324 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:23.864240 2582 scope.go:117] "RemoveContainer" containerID="d52e8afc420880b0e4e36da142e2924994e21419de33b0a04a6b22a3fd6db7fc" Jul 16 17:35:23.864399 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:23.864368 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-2wxlk_openshift-console-operator(c223b307-534a-47e4-b677-e015ec5e6861)\"" pod="openshift-console-operator/console-operator-7866497494-2wxlk" podUID="c223b307-534a-47e4-b677-e015ec5e6861" Jul 16 17:35:24.868461 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:24.868428 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kv4hq" event={"ID":"63681960-82a8-45eb-b340-adfa536295c3","Type":"ContainerStarted","Data":"7c7d3ab417fcb71966d844b4f60283ceb6a8a1d058d8fa7e41c6ccbff440ffef"} Jul 16 17:35:24.868788 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:24.868498 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kv4hq" event={"ID":"63681960-82a8-45eb-b340-adfa536295c3","Type":"ContainerStarted","Data":"79e4f5712ac9ff61352b95ef0a9727f22a386d02757b9142294296ea8e21c361"} Jul 16 17:35:24.886080 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:24.886039 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kv4hq" podStartSLOduration=3.283573452 podStartE2EDuration="5.886025544s" totalImagesPullingTime="2.602452092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:24.885032919 +0000 UTC m=+74.902262270" watchObservedRunningTime="2026-07-16 17:35:24.886025544 +0000 UTC m=+74.903254884" Jul 16 17:35:25.862952 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:25.862915 2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:25.863186 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:25.862995 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:25.863302 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:25.863290 2582 scope.go:117] "RemoveContainer" containerID="d52e8afc420880b0e4e36da142e2924994e21419de33b0a04a6b22a3fd6db7fc" Jul 16 17:35:25.863443 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:25.863423 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-2wxlk_openshift-console-operator(c223b307-534a-47e4-b677-e015ec5e6861)\"" pod="openshift-console-operator/console-operator-7866497494-2wxlk" podUID="c223b307-534a-47e4-b677-e015ec5e6861" Jul 16 17:35:26.827439 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:26.827404 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:26.827827 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:26.827444 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:26.827827 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:26.827544 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:35:26.827827 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:26.827577 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:35:26.827827 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:26.827614 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm podName:c05683ac-5e47-45a9-8de4-90bcb06edc11 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:34.827595956 +0000 UTC m=+84.844825291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hsd9q" (UID: "c05683ac-5e47-45a9-8de4-90bcb06edc11") : configmap "kube-rbac-proxy" not found Jul 16 17:35:26.827827 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:26.827632 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls podName:c05683ac-5e47-45a9-8de4-90bcb06edc11 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:34.82762341 +0000 UTC m=+84.844852729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hsd9q" (UID: "c05683ac-5e47-45a9-8de4-90bcb06edc11") : secret "insights-runtime-extractor-tls" not found Jul 16 17:35:27.129355 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:27.129279 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k4gc5\" (UID: \"edd90096-a7e6-4040-a60c-76b5030fe275\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:27.129520 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:27.129434 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 17:35:27.129592 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:27.129571 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert podName:edd90096-a7e6-4040-a60c-76b5030fe275 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:35.129549632 +0000 UTC m=+85.146778974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-k4gc5" (UID: "edd90096-a7e6-4040-a60c-76b5030fe275") : secret "networking-console-plugin-cert" not found Jul 16 17:35:31.459658 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:31.459547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:31.460038 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:31.459684 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 17:35:31.460038 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:31.459742 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls podName:b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:47.459729221 +0000 UTC m=+97.476958542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-bgl8d" (UID: "b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0") : secret "cluster-monitoring-operator-tls" not found Jul 16 17:35:34.882249 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:34.882213 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:34.882638 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:34.882255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:34.882638 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:34.882354 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:35:34.882638 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:34.882421 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls podName:c05683ac-5e47-45a9-8de4-90bcb06edc11 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:50.8824046 +0000 UTC m=+100.899633926 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hsd9q" (UID: "c05683ac-5e47-45a9-8de4-90bcb06edc11") : secret "insights-runtime-extractor-tls" not found Jul 16 17:35:34.882837 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:34.882803 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05683ac-5e47-45a9-8de4-90bcb06edc11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:35.184877 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:35.184800 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k4gc5\" (UID: \"edd90096-a7e6-4040-a60c-76b5030fe275\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:35.185000 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:35.184917 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 17:35:35.185000 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:35:35.184966 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert podName:edd90096-a7e6-4040-a60c-76b5030fe275 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:51.184953792 +0000 UTC m=+101.202183111 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-k4gc5" (UID: "edd90096-a7e6-4040-a60c-76b5030fe275") : secret "networking-console-plugin-cert" not found Jul 16 17:35:40.581547 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:40.581519 2582 scope.go:117] "RemoveContainer" containerID="d52e8afc420880b0e4e36da142e2924994e21419de33b0a04a6b22a3fd6db7fc" Jul 16 17:35:40.921219 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:40.921147 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 17:35:40.921348 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:40.921223 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-2wxlk" event={"ID":"c223b307-534a-47e4-b677-e015ec5e6861","Type":"ContainerStarted","Data":"13f75d9c84bee63806dca5333b188976c15b6f5449700cc0e7963e561b8b99fd"} Jul 16 17:35:40.921603 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:40.921574 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:40.938596 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:40.938493 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-2wxlk" podStartSLOduration=20.421026873 podStartE2EDuration="25.938458207s" totalImagesPullingTime="5.517431334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:40.938379684 +0000 UTC m=+90.955609024" watchObservedRunningTime="2026-07-16 17:35:40.938458207 +0000 UTC m=+90.955687548" Jul 16 17:35:41.322145 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:41.322118 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-2wxlk" Jul 16 17:35:47.469307 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:47.469277 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:47.471575 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:47.471557 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-bgl8d\" (UID: \"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:47.668238 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:47.668212 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" Jul 16 17:35:47.786006 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:47.785967 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d"] Jul 16 17:35:47.789362 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:35:47.789337 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ec8f32_c15f_4cdb_aefe_c0c197ba1ca0.slice/crio-59cc6017e24f724e5762dff7725aeb865e48c8bb1e3691049ba3520b500a0aa3 WatchSource:0}: Error finding container 59cc6017e24f724e5762dff7725aeb865e48c8bb1e3691049ba3520b500a0aa3: Status 404 returned error can't find the container with id 59cc6017e24f724e5762dff7725aeb865e48c8bb1e3691049ba3520b500a0aa3 Jul 16 17:35:47.942616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:47.942582 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" event={"ID":"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0","Type":"ContainerStarted","Data":"59cc6017e24f724e5762dff7725aeb865e48c8bb1e3691049ba3520b500a0aa3"} Jul 16 17:35:48.274377 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:48.274346 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:35:48.276685 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:48.276661 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59e0fc2-4364-4718-9c16-34bf2c019278-metrics-tls\") pod \"dns-default-85spg\" (UID: \"f59e0fc2-4364-4718-9c16-34bf2c019278\") " pod="openshift-dns/dns-default-85spg" Jul 16 17:35:48.575043 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:48.575019 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ntp5m\"" Jul 16 17:35:48.582447 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:48.582424 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-85spg" Jul 16 17:35:48.702526 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:48.702497 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-85spg"] Jul 16 17:35:48.705209 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:35:48.705185 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59e0fc2_4364_4718_9c16_34bf2c019278.slice/crio-ebe7252ea2881cf3aa4c9e9765cc232ea9053605667c68dc54397dcb9f99f418 WatchSource:0}: Error finding container ebe7252ea2881cf3aa4c9e9765cc232ea9053605667c68dc54397dcb9f99f418: Status 404 returned error can't find the container with id ebe7252ea2881cf3aa4c9e9765cc232ea9053605667c68dc54397dcb9f99f418 Jul 16 17:35:48.946237 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:48.946172 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-85spg" event={"ID":"f59e0fc2-4364-4718-9c16-34bf2c019278","Type":"ContainerStarted","Data":"ebe7252ea2881cf3aa4c9e9765cc232ea9053605667c68dc54397dcb9f99f418"} Jul 16 17:35:50.894259 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:50.894228 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:50.897002 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:50.896927 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05683ac-5e47-45a9-8de4-90bcb06edc11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsd9q\" (UID: \"c05683ac-5e47-45a9-8de4-90bcb06edc11\") " pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:50.954024 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:50.953995 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" event={"ID":"b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0","Type":"ContainerStarted","Data":"a3bf1635db524bc3247a613e7f523fe2b2e6dd2d876e2925ee067a3ad3051461"} Jul 16 17:35:50.973831 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:50.973777 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-bgl8d" podStartSLOduration=33.627899384 podStartE2EDuration="35.973762574s" totalImagesPullingTime="2.34586319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:50.971872738 +0000 UTC m=+100.989102082" watchObservedRunningTime="2026-07-16 17:35:50.973762574 +0000 UTC m=+100.990991914" Jul 16 17:35:51.128817 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.128789 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hsd9q" Jul 16 17:35:51.196196 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.196146 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k4gc5\" (UID: \"edd90096-a7e6-4040-a60c-76b5030fe275\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:51.199064 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.199039 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edd90096-a7e6-4040-a60c-76b5030fe275-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-k4gc5\" (UID: \"edd90096-a7e6-4040-a60c-76b5030fe275\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:51.399415 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.399377 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv"] Jul 16 17:35:51.411716 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.411655 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.420345 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.420310 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 16 17:35:51.420604 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.420581 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 16 17:35:51.420795 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.420773 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 17:35:51.421567 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.421550 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 16 17:35:51.421841 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.421819 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 16 17:35:51.421943 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.421829 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 17:35:51.423159 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.423142 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" Jul 16 17:35:51.423879 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.423865 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 17:35:51.432565 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.432542 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv"] Jul 16 17:35:51.498180 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.498142 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7d175a76-b09a-4ea5-a74f-0d12a2190209-hub\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.498346 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.498217 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d175a76-b09a-4ea5-a74f-0d12a2190209-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.498346 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.498253 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d175a76-b09a-4ea5-a74f-0d12a2190209-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.498456 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.498402 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7d175a76-b09a-4ea5-a74f-0d12a2190209-ca\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.498456 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.498437 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7d175a76-b09a-4ea5-a74f-0d12a2190209-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.498584 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.498463 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn5kc\" (UniqueName: \"kubernetes.io/projected/7d175a76-b09a-4ea5-a74f-0d12a2190209-kube-api-access-tn5kc\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.599526 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.599498 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7d175a76-b09a-4ea5-a74f-0d12a2190209-hub\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.599677 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.599570 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d175a76-b09a-4ea5-a74f-0d12a2190209-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.599677 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.599600 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d175a76-b09a-4ea5-a74f-0d12a2190209-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.599677 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.599641 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7d175a76-b09a-4ea5-a74f-0d12a2190209-ca\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.599835 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.599682 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7d175a76-b09a-4ea5-a74f-0d12a2190209-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.599835 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.599705 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn5kc\" (UniqueName: \"kubernetes.io/projected/7d175a76-b09a-4ea5-a74f-0d12a2190209-kube-api-access-tn5kc\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.600556 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.600528 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d175a76-b09a-4ea5-a74f-0d12a2190209-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.601987 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.601956 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7d175a76-b09a-4ea5-a74f-0d12a2190209-ca\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.602168 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.602151 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7d175a76-b09a-4ea5-a74f-0d12a2190209-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.602231 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.602215 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d175a76-b09a-4ea5-a74f-0d12a2190209-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.602282 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.602232 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7d175a76-b09a-4ea5-a74f-0d12a2190209-hub\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.633419 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:35:51.633397 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05683ac_5e47_45a9_8de4_90bcb06edc11.slice/crio-46f3aaed966c3941b89f4eae0a9b8562c23ecdcdd6d4367c10d1e7eb434b0f5b WatchSource:0}: Error finding container 46f3aaed966c3941b89f4eae0a9b8562c23ecdcdd6d4367c10d1e7eb434b0f5b: Status 404 returned error can't find the container with id 46f3aaed966c3941b89f4eae0a9b8562c23ecdcdd6d4367c10d1e7eb434b0f5b Jul 16 17:35:51.639795 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.639775 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hsd9q"] Jul 16 17:35:51.640138 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.640122 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn5kc\" (UniqueName: \"kubernetes.io/projected/7d175a76-b09a-4ea5-a74f-0d12a2190209-kube-api-access-tn5kc\") pod \"cluster-proxy-proxy-agent-66855cbbcd-vrwmv\" (UID: \"7d175a76-b09a-4ea5-a74f-0d12a2190209\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.672072 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.672052 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-k4gc5"] Jul 16 17:35:51.732239 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.732218 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" Jul 16 17:35:51.769913 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:35:51.769891 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd90096_a7e6_4040_a60c_76b5030fe275.slice/crio-ee45eda94c9f3904422bc809ac1f9a00a989e611de2db1f5e45aa18726ea33b1 WatchSource:0}: Error finding container ee45eda94c9f3904422bc809ac1f9a00a989e611de2db1f5e45aa18726ea33b1: Status 404 returned error can't find the container with id ee45eda94c9f3904422bc809ac1f9a00a989e611de2db1f5e45aa18726ea33b1 Jul 16 17:35:51.930059 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.930027 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv"] Jul 16 17:35:51.932143 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:35:51.932115 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d175a76_b09a_4ea5_a74f_0d12a2190209.slice/crio-72a0cb46cc6b749fa65c6d718ee91b12051ded1ae10514e603c329d6739b653b WatchSource:0}: Error finding container 72a0cb46cc6b749fa65c6d718ee91b12051ded1ae10514e603c329d6739b653b: Status 404 returned error can't find the container with id 72a0cb46cc6b749fa65c6d718ee91b12051ded1ae10514e603c329d6739b653b Jul 16 17:35:51.959529 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.959452 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-85spg" event={"ID":"f59e0fc2-4364-4718-9c16-34bf2c019278","Type":"ContainerStarted","Data":"39a0c9c0d63b0d54d768d471f32ed079bdd6442418406feadc5278b42108a350"} Jul 16 17:35:51.960658 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.960627 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" event={"ID":"edd90096-a7e6-4040-a60c-76b5030fe275","Type":"ContainerStarted","Data":"ee45eda94c9f3904422bc809ac1f9a00a989e611de2db1f5e45aa18726ea33b1"} Jul 16 17:35:51.962035 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.962012 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" event={"ID":"7d175a76-b09a-4ea5-a74f-0d12a2190209","Type":"ContainerStarted","Data":"72a0cb46cc6b749fa65c6d718ee91b12051ded1ae10514e603c329d6739b653b"} Jul 16 17:35:51.962993 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:51.962972 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hsd9q" event={"ID":"c05683ac-5e47-45a9-8de4-90bcb06edc11","Type":"ContainerStarted","Data":"46f3aaed966c3941b89f4eae0a9b8562c23ecdcdd6d4367c10d1e7eb434b0f5b"} Jul 16 17:35:52.861931 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:52.861904 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x2vn2" Jul 16 17:35:52.968893 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:52.968834 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-85spg" event={"ID":"f59e0fc2-4364-4718-9c16-34bf2c019278","Type":"ContainerStarted","Data":"1f2c270c52c8f3bc7a61bbca297ff98b1e66d609b47fe77cdcae303a3ba99e66"} Jul 16 17:35:52.969247 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:52.969085 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-85spg" Jul 16 17:35:52.987178 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:52.986973 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-85spg" podStartSLOduration=65.90028618 podStartE2EDuration="1m8.986954904s" totalImagesPullingTime="3.086668724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:52.986396949 +0000 UTC m=+103.003626292" watchObservedRunningTime="2026-07-16 17:35:52.986954904 +0000 UTC m=+103.004184248" Jul 16 17:35:54.978546 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:54.978507 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" event={"ID":"edd90096-a7e6-4040-a60c-76b5030fe275","Type":"ContainerStarted","Data":"eb7a9a35da99538613001533f58cf6e068d0a8c58d21ed45eb00930e0eb2c4c6"} Jul 16 17:35:54.998118 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:54.998063 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-k4gc5" podStartSLOduration=33.526157638 podStartE2EDuration="35.998043485s" totalImagesPullingTime="2.471885847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:54.997593147 +0000 UTC m=+105.014822489" watchObservedRunningTime="2026-07-16 17:35:54.998043485 +0000 UTC m=+105.015272821" Jul 16 17:35:59.997018 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:59.996984 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" event={"ID":"7d175a76-b09a-4ea5-a74f-0d12a2190209","Type":"ContainerStarted","Data":"a949b384e260e0c11d8ba266c801fe91d44fe007329fa51e7b782c33645efd8d"} Jul 16 17:35:59.997413 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:59.997034 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" event={"ID":"7d175a76-b09a-4ea5-a74f-0d12a2190209","Type":"ContainerStarted","Data":"b3ed308e1fd2f0d55c01ca1536b7a22f3a30217fa320d09d8ee1d0d97e5e43d7"} Jul 16 17:35:59.997413 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:59.997048 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" event={"ID":"7d175a76-b09a-4ea5-a74f-0d12a2190209","Type":"ContainerStarted","Data":"6e639f09ead7fddc46e49e5644af85b34facb12b0297093070ab17fb80a0fd24"} Jul 16 17:35:59.998372 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:35:59.998354 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hsd9q" event={"ID":"c05683ac-5e47-45a9-8de4-90bcb06edc11","Type":"ContainerStarted","Data":"23b4783a0ba45a415433d19d911a8bceed8cfe44ad5b821980ee13d876954024"} Jul 16 17:36:00.020599 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:00.020556 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66855cbbcd-vrwmv" podStartSLOduration=1.282287917 podStartE2EDuration="9.020541759s" totalImagesPullingTime="7.738253842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:00.018674514 +0000 UTC m=+110.035903877" watchObservedRunningTime="2026-07-16 17:36:00.020541759 +0000 UTC m=+110.037771139" Jul 16 17:36:02.975170 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:02.975133 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-85spg" Jul 16 17:36:04.015098 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:04.015059 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hsd9q" event={"ID":"c05683ac-5e47-45a9-8de4-90bcb06edc11","Type":"ContainerStarted","Data":"69e7ce99ef9d3bcd35448a2bc02695633f336f19ac6420dec92db41a0d0bfd9d"} Jul 16 17:36:07.028193 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:07.028155 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hsd9q" event={"ID":"c05683ac-5e47-45a9-8de4-90bcb06edc11","Type":"ContainerStarted","Data":"d55424a40bd3a46808f553ee47e080cc62a2d9e0fceab72792f9223deae155ab"} Jul 16 17:36:07.048776 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:07.048727 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hsd9q" podStartSLOduration=34.381725791 podStartE2EDuration="49.04871432s" totalImagesPullingTime="14.666988529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:35:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:07.047198333 +0000 UTC m=+117.064427678" watchObservedRunningTime="2026-07-16 17:36:07.04871432 +0000 UTC m=+117.065943660" Jul 16 17:36:16.185734 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.185705 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vqkkm"] Jul 16 17:36:16.187946 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.187928 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.190378 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.190269 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 17:36:16.190378 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.190281 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 17:36:16.190378 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.190281 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zrzhn\"" Jul 16 17:36:16.190715 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.190637 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 17:36:16.191584 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.191568 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 17:36:16.285979 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.285957 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-tls\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.286071 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.285984 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.286071 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.286007 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4q5w\" (UniqueName: \"kubernetes.io/projected/44374839-36f1-4dcd-8e22-3280caf48704-kube-api-access-c4q5w\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.286071 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.286035 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-textfile\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.286071 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.286058 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44374839-36f1-4dcd-8e22-3280caf48704-sys\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.286227 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.286085 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-wtmp\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.286227 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.286102 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44374839-36f1-4dcd-8e22-3280caf48704-metrics-client-ca\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.286227 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.286155 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44374839-36f1-4dcd-8e22-3280caf48704-root\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.286227 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.286176 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.386464 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.386439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44374839-36f1-4dcd-8e22-3280caf48704-sys\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.386464 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.386480 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-wtmp\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.386633 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.386499 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44374839-36f1-4dcd-8e22-3280caf48704-metrics-client-ca\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.386633 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.386525 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44374839-36f1-4dcd-8e22-3280caf48704-root\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.386633 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.386551 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.386633 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.386556 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44374839-36f1-4dcd-8e22-3280caf48704-sys\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.386633 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.386595 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44374839-36f1-4dcd-8e22-3280caf48704-root\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.386633 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.386620 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-wtmp\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.386832 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.386636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-tls\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.386832 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.386661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.386832 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.386685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4q5w\" (UniqueName: \"kubernetes.io/projected/44374839-36f1-4dcd-8e22-3280caf48704-kube-api-access-c4q5w\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.386832 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.386713 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-textfile\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.387094 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.387065 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-textfile\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.387218 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.387195 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44374839-36f1-4dcd-8e22-3280caf48704-metrics-client-ca\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.387307 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.387274 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.389515 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.389495 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.389618 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.389601 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44374839-36f1-4dcd-8e22-3280caf48704-node-exporter-tls\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.398326 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.398301 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4q5w\" (UniqueName: \"kubernetes.io/projected/44374839-36f1-4dcd-8e22-3280caf48704-kube-api-access-c4q5w\") pod \"node-exporter-vqkkm\" (UID: \"44374839-36f1-4dcd-8e22-3280caf48704\") " pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.496829 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:16.496801 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vqkkm" Jul 16 17:36:16.506289 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:36:16.506260 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44374839_36f1_4dcd_8e22_3280caf48704.slice/crio-7e615b3b828a264a5acee2e46fa5ecb7f90075c80971339318de5425e47e6362 WatchSource:0}: Error finding container 7e615b3b828a264a5acee2e46fa5ecb7f90075c80971339318de5425e47e6362: Status 404 returned error can't find the container with id 7e615b3b828a264a5acee2e46fa5ecb7f90075c80971339318de5425e47e6362 Jul 16 17:36:17.061929 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:17.061895 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqkkm" event={"ID":"44374839-36f1-4dcd-8e22-3280caf48704","Type":"ContainerStarted","Data":"7e615b3b828a264a5acee2e46fa5ecb7f90075c80971339318de5425e47e6362"} Jul 16 17:36:18.066875 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:18.066841 2582 generic.go:356] "Generic (PLEG): container finished" podID="44374839-36f1-4dcd-8e22-3280caf48704" containerID="d9f8c7ca9b8200ca65c1fefdc9baf49714efbacf3335e6c2cdbe0d2e6cb1e7e1" exitCode=0 Jul 16 17:36:18.067228 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:18.066903 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqkkm" event={"ID":"44374839-36f1-4dcd-8e22-3280caf48704","Type":"ContainerDied","Data":"d9f8c7ca9b8200ca65c1fefdc9baf49714efbacf3335e6c2cdbe0d2e6cb1e7e1"} Jul 16 17:36:19.071849 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:19.071805 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqkkm" event={"ID":"44374839-36f1-4dcd-8e22-3280caf48704","Type":"ContainerStarted","Data":"8af880f4ce702ac6980b46981d5f3754495b9056bd8c8c3d35831e54f78a6711"} Jul 16 17:36:19.072250 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:19.071874 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqkkm" event={"ID":"44374839-36f1-4dcd-8e22-3280caf48704","Type":"ContainerStarted","Data":"8cafcaabc5be90ceb2209f7105cec0b9b43c9236094e9e2c40d49cb3c35b32f3"} Jul 16 17:36:19.094042 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:19.093995 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vqkkm" podStartSLOduration=2.098321524 podStartE2EDuration="3.093980581s" totalImagesPullingTime="927.777056ms" totalInitContainerRuntime="67.882001ms" isStatefulPod=true podCreationTimestamp="2026-07-16 17:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:19.092954387 +0000 UTC m=+129.110183752" watchObservedRunningTime="2026-07-16 17:36:19.093980581 +0000 UTC m=+129.111209952" Jul 16 17:36:20.214805 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:20.214766 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:36:20.217128 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:20.217109 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc349e33-d51c-40bf-8890-392c3393299d-metrics-certs\") pod \"network-metrics-daemon-5qrcw\" (UID: \"dc349e33-d51c-40bf-8890-392c3393299d\") " pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:36:20.301746 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:20.301716 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p9hcv\"" Jul 16 17:36:20.309742 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:20.309722 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qrcw" Jul 16 17:36:20.459271 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:20.459243 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qrcw"] Jul 16 17:36:20.462665 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:36:20.462637 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc349e33_d51c_40bf_8890_392c3393299d.slice/crio-3c6a3d952cb64af21b2c01294fb86af74c7f498283bd2571e3af4285d10c350a WatchSource:0}: Error finding container 3c6a3d952cb64af21b2c01294fb86af74c7f498283bd2571e3af4285d10c350a: Status 404 returned error can't find the container with id 3c6a3d952cb64af21b2c01294fb86af74c7f498283bd2571e3af4285d10c350a Jul 16 17:36:21.080808 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:21.079776 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qrcw" event={"ID":"dc349e33-d51c-40bf-8890-392c3393299d","Type":"ContainerStarted","Data":"3c6a3d952cb64af21b2c01294fb86af74c7f498283bd2571e3af4285d10c350a"} Jul 16 17:36:22.084590 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:22.084552 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qrcw" event={"ID":"dc349e33-d51c-40bf-8890-392c3393299d","Type":"ContainerStarted","Data":"f8ad87a743987d1540bdf07a53a86f0e17ade079a43b495b4155ec7bb6f33667"} Jul 16 17:36:22.084950 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:22.084602 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qrcw" event={"ID":"dc349e33-d51c-40bf-8890-392c3393299d","Type":"ContainerStarted","Data":"ec6773d2f3af5dacc2332b2b11b8de936079faaacc27d8d8bb8dee7498898078"} Jul 16 17:36:22.103514 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:22.103439 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5qrcw" podStartSLOduration=130.69399368 podStartE2EDuration="2m12.103423666s" totalImagesPullingTime="1.409429986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:22.101989074 +0000 UTC m=+132.119218416" watchObservedRunningTime="2026-07-16 17:36:22.103423666 +0000 UTC m=+132.120653007" Jul 16 17:36:47.177034 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:47.177003 2582 generic.go:356] "Generic (PLEG): container finished" podID="2ec09f0e-e2cc-4e21-9200-1b91486a5251" containerID="ad99ad66b034d1c0ce2ab81c662a35a2811b1b96e674004e9eeb189994bcc5ca" exitCode=0 Jul 16 17:36:47.177359 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:47.177072 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" event={"ID":"2ec09f0e-e2cc-4e21-9200-1b91486a5251","Type":"ContainerDied","Data":"ad99ad66b034d1c0ce2ab81c662a35a2811b1b96e674004e9eeb189994bcc5ca"} Jul 16 17:36:47.177359 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:47.177352 2582 scope.go:117] "RemoveContainer" containerID="ad99ad66b034d1c0ce2ab81c662a35a2811b1b96e674004e9eeb189994bcc5ca" Jul 16 17:36:48.182581 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:36:48.182546 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rf6rf" event={"ID":"2ec09f0e-e2cc-4e21-9200-1b91486a5251","Type":"ContainerStarted","Data":"17be6f5b424df7e0da672d488f6891b2d890b2a42cf62f39440828f05bad54b4"} Jul 16 17:39:10.381948 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:39:10.381919 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 17:39:10.383514 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:39:10.383492 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 17:39:10.389349 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:39:10.389319 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 17:39:10.390539 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:39:10.390519 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 17:39:10.393295 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:39:10.393276 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 17:41:10.415777 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.415743 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9"] Jul 16 17:41:10.417705 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.417690 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" Jul 16 17:41:10.420256 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.420238 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 16 17:41:10.421224 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.421204 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xx5gn\"" Jul 16 17:41:10.421321 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.421227 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 16 17:41:10.421321 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.421208 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 16 17:41:10.421321 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.421263 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Jul 16 17:41:10.421321 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.421297 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 16 17:41:10.432837 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.432815 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9"] Jul 16 17:41:10.497488 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.497445 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a2c02ec5-5a49-40f5-9f97-9165543f8954-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tbgs9\" (UID: \"a2c02ec5-5a49-40f5-9f97-9165543f8954\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" Jul 16 17:41:10.497588 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.497510 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5ftb\" (UniqueName: \"kubernetes.io/projected/a2c02ec5-5a49-40f5-9f97-9165543f8954-kube-api-access-k5ftb\") pod \"keda-metrics-apiserver-55557df8c-tbgs9\" (UID: \"a2c02ec5-5a49-40f5-9f97-9165543f8954\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" Jul 16 17:41:10.497588 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.497548 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a2c02ec5-5a49-40f5-9f97-9165543f8954-certificates\") pod \"keda-metrics-apiserver-55557df8c-tbgs9\" (UID: \"a2c02ec5-5a49-40f5-9f97-9165543f8954\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" Jul 16 17:41:10.598688 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.598663 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5ftb\" (UniqueName: \"kubernetes.io/projected/a2c02ec5-5a49-40f5-9f97-9165543f8954-kube-api-access-k5ftb\") pod \"keda-metrics-apiserver-55557df8c-tbgs9\" (UID: \"a2c02ec5-5a49-40f5-9f97-9165543f8954\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" Jul 16 17:41:10.598785 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.598694 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a2c02ec5-5a49-40f5-9f97-9165543f8954-certificates\") pod \"keda-metrics-apiserver-55557df8c-tbgs9\" (UID: \"a2c02ec5-5a49-40f5-9f97-9165543f8954\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" Jul 16 17:41:10.598785 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.598727 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a2c02ec5-5a49-40f5-9f97-9165543f8954-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tbgs9\" (UID: \"a2c02ec5-5a49-40f5-9f97-9165543f8954\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" Jul 16 17:41:10.599039 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.599023 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a2c02ec5-5a49-40f5-9f97-9165543f8954-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tbgs9\" (UID: \"a2c02ec5-5a49-40f5-9f97-9165543f8954\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" Jul 16 17:41:10.600966 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.600947 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a2c02ec5-5a49-40f5-9f97-9165543f8954-certificates\") pod \"keda-metrics-apiserver-55557df8c-tbgs9\" (UID: \"a2c02ec5-5a49-40f5-9f97-9165543f8954\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" Jul 16 17:41:10.609787 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.609769 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5ftb\" (UniqueName: \"kubernetes.io/projected/a2c02ec5-5a49-40f5-9f97-9165543f8954-kube-api-access-k5ftb\") pod \"keda-metrics-apiserver-55557df8c-tbgs9\" (UID: \"a2c02ec5-5a49-40f5-9f97-9165543f8954\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" Jul 16 17:41:10.727639 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.727614 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" Jul 16 17:41:10.854023 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.853915 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9"] Jul 16 17:41:10.856700 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:41:10.856671 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c02ec5_5a49_40f5_9f97_9165543f8954.slice/crio-9818c5a79de4192f2fbcc57fe3165afde38dd874aed51495d6cfe174191e1600 WatchSource:0}: Error finding container 9818c5a79de4192f2fbcc57fe3165afde38dd874aed51495d6cfe174191e1600: Status 404 returned error can't find the container with id 9818c5a79de4192f2fbcc57fe3165afde38dd874aed51495d6cfe174191e1600 Jul 16 17:41:10.858405 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:10.858388 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:41:11.069759 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:11.069698 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" event={"ID":"a2c02ec5-5a49-40f5-9f97-9165543f8954","Type":"ContainerStarted","Data":"9818c5a79de4192f2fbcc57fe3165afde38dd874aed51495d6cfe174191e1600"} Jul 16 17:41:14.083356 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:14.083327 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" event={"ID":"a2c02ec5-5a49-40f5-9f97-9165543f8954","Type":"ContainerStarted","Data":"9fbfe22f02df67b444160a9394bfc3b6602d050c294ea510a3da41023f29618f"} Jul 16 17:41:14.083720 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:14.083455 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" Jul 16 17:41:14.101614 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:14.101572 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" podStartSLOduration=1.358378112 podStartE2EDuration="4.101559692s" totalImagesPullingTime="2.74318158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:14.101111468 +0000 UTC m=+424.118340811" watchObservedRunningTime="2026-07-16 17:41:14.101559692 +0000 UTC m=+424.118789033" Jul 16 17:41:25.091258 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:41:25.091224 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tbgs9" Jul 16 17:42:17.562486 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.562442 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-576d7b7757-tvrtp"] Jul 16 17:42:17.564643 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.564627 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" Jul 16 17:42:17.567582 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.567560 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 16 17:42:17.567770 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.567756 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Jul 16 17:42:17.568626 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.568606 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-5qm24\"" Jul 16 17:42:17.568728 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.568635 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 16 17:42:17.574195 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.574175 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-576d7b7757-tvrtp"] Jul 16 17:42:17.642681 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.642656 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b1ca299-564b-4a77-8814-b4ba2399938a-cert\") pod \"kserve-controller-manager-576d7b7757-tvrtp\" (UID: \"3b1ca299-564b-4a77-8814-b4ba2399938a\") " pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" Jul 16 17:42:17.642778 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.642706 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmrd6\" (UniqueName: \"kubernetes.io/projected/3b1ca299-564b-4a77-8814-b4ba2399938a-kube-api-access-bmrd6\") pod \"kserve-controller-manager-576d7b7757-tvrtp\" (UID: \"3b1ca299-564b-4a77-8814-b4ba2399938a\") " pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" Jul 16 17:42:17.743305 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.743274 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmrd6\" (UniqueName: \"kubernetes.io/projected/3b1ca299-564b-4a77-8814-b4ba2399938a-kube-api-access-bmrd6\") pod \"kserve-controller-manager-576d7b7757-tvrtp\" (UID: \"3b1ca299-564b-4a77-8814-b4ba2399938a\") " pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" Jul 16 17:42:17.743414 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.743327 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b1ca299-564b-4a77-8814-b4ba2399938a-cert\") pod \"kserve-controller-manager-576d7b7757-tvrtp\" (UID: \"3b1ca299-564b-4a77-8814-b4ba2399938a\") " pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" Jul 16 17:42:17.745784 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.745764 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b1ca299-564b-4a77-8814-b4ba2399938a-cert\") pod \"kserve-controller-manager-576d7b7757-tvrtp\" (UID: \"3b1ca299-564b-4a77-8814-b4ba2399938a\") " pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" Jul 16 17:42:17.752123 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.752095 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmrd6\" (UniqueName: \"kubernetes.io/projected/3b1ca299-564b-4a77-8814-b4ba2399938a-kube-api-access-bmrd6\") pod \"kserve-controller-manager-576d7b7757-tvrtp\" (UID: \"3b1ca299-564b-4a77-8814-b4ba2399938a\") " pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" Jul 16 17:42:17.874544 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.874493 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" Jul 16 17:42:17.997455 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:17.997424 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-576d7b7757-tvrtp"] Jul 16 17:42:18.000728 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:42:18.000693 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1ca299_564b_4a77_8814_b4ba2399938a.slice/crio-ad172d806002d8bea7e784c08522ac44e6c3cef589b82dce1e8ebeda1ed1b309 WatchSource:0}: Error finding container ad172d806002d8bea7e784c08522ac44e6c3cef589b82dce1e8ebeda1ed1b309: Status 404 returned error can't find the container with id ad172d806002d8bea7e784c08522ac44e6c3cef589b82dce1e8ebeda1ed1b309 Jul 16 17:42:18.306577 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:18.306536 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" event={"ID":"3b1ca299-564b-4a77-8814-b4ba2399938a","Type":"ContainerStarted","Data":"ad172d806002d8bea7e784c08522ac44e6c3cef589b82dce1e8ebeda1ed1b309"} Jul 16 17:42:21.320076 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:21.320030 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" event={"ID":"3b1ca299-564b-4a77-8814-b4ba2399938a","Type":"ContainerStarted","Data":"29243215893a59ccf5e27e36825e8176651d92ed8151d58493bc02e31fe0a289"} Jul 16 17:42:21.320507 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:21.320144 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" Jul 16 17:42:21.339169 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:21.339120 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" podStartSLOduration=2.040557008 podStartE2EDuration="4.339103812s" totalImagesPullingTime="2.298546804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:42:21.337189423 +0000 UTC m=+491.354418765" watchObservedRunningTime="2026-07-16 17:42:21.339103812 +0000 UTC m=+491.356333155" Jul 16 17:42:52.329569 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:52.329539 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" Jul 16 17:42:59.109966 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.109931 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-576d7b7757-tvrtp"] Jul 16 17:42:59.110381 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.110112 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" podUID="3b1ca299-564b-4a77-8814-b4ba2399938a" containerName="manager" containerID="cri-o://29243215893a59ccf5e27e36825e8176651d92ed8151d58493bc02e31fe0a289" gracePeriod=10 Jul 16 17:42:59.132258 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.132234 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-576d7b7757-5rqdx"] Jul 16 17:42:59.134547 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.134529 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-576d7b7757-5rqdx" Jul 16 17:42:59.145267 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.145242 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-576d7b7757-5rqdx"] Jul 16 17:42:59.226286 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.226262 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2m6r\" (UniqueName: \"kubernetes.io/projected/0ee5d930-4121-453b-847d-fb6edb7ba63f-kube-api-access-p2m6r\") pod \"kserve-controller-manager-576d7b7757-5rqdx\" (UID: \"0ee5d930-4121-453b-847d-fb6edb7ba63f\") " pod="kserve/kserve-controller-manager-576d7b7757-5rqdx" Jul 16 17:42:59.226370 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.226309 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ee5d930-4121-453b-847d-fb6edb7ba63f-cert\") pod \"kserve-controller-manager-576d7b7757-5rqdx\" (UID: \"0ee5d930-4121-453b-847d-fb6edb7ba63f\") " pod="kserve/kserve-controller-manager-576d7b7757-5rqdx" Jul 16 17:42:59.327641 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.327612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2m6r\" (UniqueName: \"kubernetes.io/projected/0ee5d930-4121-453b-847d-fb6edb7ba63f-kube-api-access-p2m6r\") pod \"kserve-controller-manager-576d7b7757-5rqdx\" (UID: \"0ee5d930-4121-453b-847d-fb6edb7ba63f\") " pod="kserve/kserve-controller-manager-576d7b7757-5rqdx" Jul 16 17:42:59.327782 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.327651 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ee5d930-4121-453b-847d-fb6edb7ba63f-cert\") pod \"kserve-controller-manager-576d7b7757-5rqdx\" (UID: \"0ee5d930-4121-453b-847d-fb6edb7ba63f\") " pod="kserve/kserve-controller-manager-576d7b7757-5rqdx" Jul 16 17:42:59.330216 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.330191 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ee5d930-4121-453b-847d-fb6edb7ba63f-cert\") pod \"kserve-controller-manager-576d7b7757-5rqdx\" (UID: \"0ee5d930-4121-453b-847d-fb6edb7ba63f\") " pod="kserve/kserve-controller-manager-576d7b7757-5rqdx" Jul 16 17:42:59.336679 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.336659 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2m6r\" (UniqueName: \"kubernetes.io/projected/0ee5d930-4121-453b-847d-fb6edb7ba63f-kube-api-access-p2m6r\") pod \"kserve-controller-manager-576d7b7757-5rqdx\" (UID: \"0ee5d930-4121-453b-847d-fb6edb7ba63f\") " pod="kserve/kserve-controller-manager-576d7b7757-5rqdx" Jul 16 17:42:59.350914 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.350895 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" Jul 16 17:42:59.428404 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.428324 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmrd6\" (UniqueName: \"kubernetes.io/projected/3b1ca299-564b-4a77-8814-b4ba2399938a-kube-api-access-bmrd6\") pod \"3b1ca299-564b-4a77-8814-b4ba2399938a\" (UID: \"3b1ca299-564b-4a77-8814-b4ba2399938a\") " Jul 16 17:42:59.428588 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.428406 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b1ca299-564b-4a77-8814-b4ba2399938a-cert\") pod \"3b1ca299-564b-4a77-8814-b4ba2399938a\" (UID: \"3b1ca299-564b-4a77-8814-b4ba2399938a\") " Jul 16 17:42:59.430602 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.430573 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1ca299-564b-4a77-8814-b4ba2399938a-cert" (OuterVolumeSpecName: "cert") pod "3b1ca299-564b-4a77-8814-b4ba2399938a" (UID: "3b1ca299-564b-4a77-8814-b4ba2399938a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:42:59.430698 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.430679 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1ca299-564b-4a77-8814-b4ba2399938a-kube-api-access-bmrd6" (OuterVolumeSpecName: "kube-api-access-bmrd6") pod "3b1ca299-564b-4a77-8814-b4ba2399938a" (UID: "3b1ca299-564b-4a77-8814-b4ba2399938a"). InnerVolumeSpecName "kube-api-access-bmrd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:42:59.453637 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.453610 2582 generic.go:356] "Generic (PLEG): container finished" podID="3b1ca299-564b-4a77-8814-b4ba2399938a" containerID="29243215893a59ccf5e27e36825e8176651d92ed8151d58493bc02e31fe0a289" exitCode=0 Jul 16 17:42:59.453733 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.453654 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" Jul 16 17:42:59.453733 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.453692 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" event={"ID":"3b1ca299-564b-4a77-8814-b4ba2399938a","Type":"ContainerDied","Data":"29243215893a59ccf5e27e36825e8176651d92ed8151d58493bc02e31fe0a289"} Jul 16 17:42:59.453800 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.453746 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-576d7b7757-tvrtp" event={"ID":"3b1ca299-564b-4a77-8814-b4ba2399938a","Type":"ContainerDied","Data":"ad172d806002d8bea7e784c08522ac44e6c3cef589b82dce1e8ebeda1ed1b309"} Jul 16 17:42:59.453800 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.453779 2582 scope.go:117] "RemoveContainer" containerID="29243215893a59ccf5e27e36825e8176651d92ed8151d58493bc02e31fe0a289" Jul 16 17:42:59.462147 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.462130 2582 scope.go:117] "RemoveContainer" containerID="29243215893a59ccf5e27e36825e8176651d92ed8151d58493bc02e31fe0a289" Jul 16 17:42:59.462422 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:42:59.462397 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29243215893a59ccf5e27e36825e8176651d92ed8151d58493bc02e31fe0a289\": container with ID starting with 29243215893a59ccf5e27e36825e8176651d92ed8151d58493bc02e31fe0a289 not found: ID does not exist" containerID="29243215893a59ccf5e27e36825e8176651d92ed8151d58493bc02e31fe0a289" Jul 16 17:42:59.462515 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.462432 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29243215893a59ccf5e27e36825e8176651d92ed8151d58493bc02e31fe0a289"} err="failed to get container status \"29243215893a59ccf5e27e36825e8176651d92ed8151d58493bc02e31fe0a289\": rpc error: code = NotFound desc = could not find container \"29243215893a59ccf5e27e36825e8176651d92ed8151d58493bc02e31fe0a289\": container with ID starting with 29243215893a59ccf5e27e36825e8176651d92ed8151d58493bc02e31fe0a289 not found: ID does not exist" Jul 16 17:42:59.476102 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.476081 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-576d7b7757-tvrtp"] Jul 16 17:42:59.480370 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.480351 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-576d7b7757-tvrtp"] Jul 16 17:42:59.490987 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.490967 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-576d7b7757-5rqdx" Jul 16 17:42:59.529774 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.529740 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmrd6\" (UniqueName: \"kubernetes.io/projected/3b1ca299-564b-4a77-8814-b4ba2399938a-kube-api-access-bmrd6\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:42:59.529774 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.529779 2582 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b1ca299-564b-4a77-8814-b4ba2399938a-cert\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:42:59.613727 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:42:59.613584 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-576d7b7757-5rqdx"] Jul 16 17:42:59.616253 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:42:59.616220 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee5d930_4121_453b_847d_fb6edb7ba63f.slice/crio-0536dd3d1b7542b6f60a98815301a4d515eb813f42b3f68994cf13b1368cc72f WatchSource:0}: Error finding container 0536dd3d1b7542b6f60a98815301a4d515eb813f42b3f68994cf13b1368cc72f: Status 404 returned error can't find the container with id 0536dd3d1b7542b6f60a98815301a4d515eb813f42b3f68994cf13b1368cc72f Jul 16 17:43:00.460002 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:00.459919 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-576d7b7757-5rqdx" event={"ID":"0ee5d930-4121-453b-847d-fb6edb7ba63f","Type":"ContainerStarted","Data":"8447bc8432b6a26350176896418303850e1b09e9cf2128b9015dfc0295b3e795"} Jul 16 17:43:00.460002 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:00.459980 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-576d7b7757-5rqdx" event={"ID":"0ee5d930-4121-453b-847d-fb6edb7ba63f","Type":"ContainerStarted","Data":"0536dd3d1b7542b6f60a98815301a4d515eb813f42b3f68994cf13b1368cc72f"} Jul 16 17:43:00.460396 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:00.460012 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-576d7b7757-5rqdx" Jul 16 17:43:00.477510 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:00.477439 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-576d7b7757-5rqdx" podStartSLOduration=1.032282091 podStartE2EDuration="1.477423305s" totalImagesPullingTime="445.141214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:43:00.476114124 +0000 UTC m=+530.493343465" watchObservedRunningTime="2026-07-16 17:43:00.477423305 +0000 UTC m=+530.494652646" Jul 16 17:43:00.584169 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:00.584103 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b1ca299-564b-4a77-8814-b4ba2399938a" path="/var/lib/kubelet/pods/3b1ca299-564b-4a77-8814-b4ba2399938a/volumes" Jul 16 17:43:31.468402 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:31.468371 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-576d7b7757-5rqdx" Jul 16 17:43:32.524896 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:32.524850 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-vmwwk"] Jul 16 17:43:32.525337 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:32.525281 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1ca299-564b-4a77-8814-b4ba2399938a" containerName="manager" Jul 16 17:43:32.525337 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:32.525300 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1ca299-564b-4a77-8814-b4ba2399938a" containerName="manager" Jul 16 17:43:32.525448 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:32.525365 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1ca299-564b-4a77-8814-b4ba2399938a" containerName="manager" Jul 16 17:43:32.528322 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:32.528302 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-vmwwk" Jul 16 17:43:32.530963 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:32.530948 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-vghvf\"" Jul 16 17:43:32.531040 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:32.530950 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Jul 16 17:43:32.538198 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:32.538173 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-vmwwk"] Jul 16 17:43:32.661761 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:32.661731 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5fvf\" (UniqueName: \"kubernetes.io/projected/d80d490e-605e-4b33-9338-304ed98b3c60-kube-api-access-n5fvf\") pod \"model-serving-api-fd65d7d6-vmwwk\" (UID: \"d80d490e-605e-4b33-9338-304ed98b3c60\") " pod="kserve/model-serving-api-fd65d7d6-vmwwk" Jul 16 17:43:32.661893 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:32.661766 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d80d490e-605e-4b33-9338-304ed98b3c60-tls-certs\") pod \"model-serving-api-fd65d7d6-vmwwk\" (UID: \"d80d490e-605e-4b33-9338-304ed98b3c60\") " pod="kserve/model-serving-api-fd65d7d6-vmwwk" Jul 16 17:43:32.762984 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:32.762956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5fvf\" (UniqueName: \"kubernetes.io/projected/d80d490e-605e-4b33-9338-304ed98b3c60-kube-api-access-n5fvf\") pod \"model-serving-api-fd65d7d6-vmwwk\" (UID: \"d80d490e-605e-4b33-9338-304ed98b3c60\") " pod="kserve/model-serving-api-fd65d7d6-vmwwk" Jul 16 17:43:32.762984 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:32.762988 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d80d490e-605e-4b33-9338-304ed98b3c60-tls-certs\") pod \"model-serving-api-fd65d7d6-vmwwk\" (UID: \"d80d490e-605e-4b33-9338-304ed98b3c60\") " pod="kserve/model-serving-api-fd65d7d6-vmwwk" Jul 16 17:43:32.763199 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:43:32.763111 2582 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 16 17:43:32.763199 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:43:32.763176 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80d490e-605e-4b33-9338-304ed98b3c60-tls-certs podName:d80d490e-605e-4b33-9338-304ed98b3c60 nodeName:}" failed. No retries permitted until 2026-07-16 17:43:33.263156672 +0000 UTC m=+563.280385993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d80d490e-605e-4b33-9338-304ed98b3c60-tls-certs") pod "model-serving-api-fd65d7d6-vmwwk" (UID: "d80d490e-605e-4b33-9338-304ed98b3c60") : secret "model-serving-api-tls" not found Jul 16 17:43:32.776161 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:32.776100 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5fvf\" (UniqueName: \"kubernetes.io/projected/d80d490e-605e-4b33-9338-304ed98b3c60-kube-api-access-n5fvf\") pod \"model-serving-api-fd65d7d6-vmwwk\" (UID: \"d80d490e-605e-4b33-9338-304ed98b3c60\") " pod="kserve/model-serving-api-fd65d7d6-vmwwk" Jul 16 17:43:33.265966 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:33.265939 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d80d490e-605e-4b33-9338-304ed98b3c60-tls-certs\") pod \"model-serving-api-fd65d7d6-vmwwk\" (UID: \"d80d490e-605e-4b33-9338-304ed98b3c60\") " pod="kserve/model-serving-api-fd65d7d6-vmwwk" Jul 16 17:43:33.268533 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:33.268509 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d80d490e-605e-4b33-9338-304ed98b3c60-tls-certs\") pod \"model-serving-api-fd65d7d6-vmwwk\" (UID: \"d80d490e-605e-4b33-9338-304ed98b3c60\") " pod="kserve/model-serving-api-fd65d7d6-vmwwk" Jul 16 17:43:33.438552 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:33.438521 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-vmwwk" Jul 16 17:43:33.572180 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:33.572154 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-vmwwk"] Jul 16 17:43:33.572778 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:43:33.572752 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80d490e_605e_4b33_9338_304ed98b3c60.slice/crio-18f2760b7b71f1f7cb018016b075b245d1ed6f7bbfd841bcfb6d2046f025986c WatchSource:0}: Error finding container 18f2760b7b71f1f7cb018016b075b245d1ed6f7bbfd841bcfb6d2046f025986c: Status 404 returned error can't find the container with id 18f2760b7b71f1f7cb018016b075b245d1ed6f7bbfd841bcfb6d2046f025986c Jul 16 17:43:33.578081 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:33.578055 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-vmwwk" event={"ID":"d80d490e-605e-4b33-9338-304ed98b3c60","Type":"ContainerStarted","Data":"18f2760b7b71f1f7cb018016b075b245d1ed6f7bbfd841bcfb6d2046f025986c"} Jul 16 17:43:35.586909 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:35.586874 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-vmwwk" event={"ID":"d80d490e-605e-4b33-9338-304ed98b3c60","Type":"ContainerStarted","Data":"a83bd5365b22e092827afb7f0051c0e18e818da250d5226d384da3e9cfd0a95c"} Jul 16 17:43:35.587259 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:35.586982 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-vmwwk" Jul 16 17:43:35.606982 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:35.606942 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-vmwwk" podStartSLOduration=2.088986387 podStartE2EDuration="3.6069287s" totalImagesPullingTime="1.517942313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:43:35.605492618 +0000 UTC m=+565.622721966" watchObservedRunningTime="2026-07-16 17:43:35.6069287 +0000 UTC m=+565.624158041" Jul 16 17:43:46.594280 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:43:46.594246 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-vmwwk" Jul 16 17:44:10.406995 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:44:10.406965 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 17:44:10.407581 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:44:10.407218 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 17:44:10.413158 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:44:10.413133 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 17:44:10.413256 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:44:10.413181 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 17:45:16.458446 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.458406 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj"] Jul 16 17:45:16.461625 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.461610 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:16.464134 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.464110 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wv2xq\"" Jul 16 17:45:16.464273 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.464152 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e62f1-kube-rbac-proxy-sar-config\"" Jul 16 17:45:16.464273 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.464159 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e62f1-predictor-serving-cert\"" Jul 16 17:45:16.465011 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.464992 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 16 17:45:16.465180 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.465073 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 16 17:45:16.472107 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.472087 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj"] Jul 16 17:45:16.554395 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.554365 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdgn\" (UniqueName: \"kubernetes.io/projected/1161db69-91a5-43f1-b15b-abe28234f18c-kube-api-access-7pdgn\") pod \"success-200-isvc-e62f1-predictor-7d46c6569f-w87lj\" (UID: \"1161db69-91a5-43f1-b15b-abe28234f18c\") " pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:16.554564 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.554401 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1161db69-91a5-43f1-b15b-abe28234f18c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e62f1-predictor-7d46c6569f-w87lj\" (UID: \"1161db69-91a5-43f1-b15b-abe28234f18c\") " pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:16.554564 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.554435 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1161db69-91a5-43f1-b15b-abe28234f18c-proxy-tls\") pod \"success-200-isvc-e62f1-predictor-7d46c6569f-w87lj\" (UID: \"1161db69-91a5-43f1-b15b-abe28234f18c\") " pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:16.655151 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.655126 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdgn\" (UniqueName: \"kubernetes.io/projected/1161db69-91a5-43f1-b15b-abe28234f18c-kube-api-access-7pdgn\") pod \"success-200-isvc-e62f1-predictor-7d46c6569f-w87lj\" (UID: \"1161db69-91a5-43f1-b15b-abe28234f18c\") " pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:16.655151 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.655154 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1161db69-91a5-43f1-b15b-abe28234f18c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e62f1-predictor-7d46c6569f-w87lj\" (UID: \"1161db69-91a5-43f1-b15b-abe28234f18c\") " pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:16.655342 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.655208 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1161db69-91a5-43f1-b15b-abe28234f18c-proxy-tls\") pod \"success-200-isvc-e62f1-predictor-7d46c6569f-w87lj\" (UID: \"1161db69-91a5-43f1-b15b-abe28234f18c\") " pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:16.655795 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.655773 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1161db69-91a5-43f1-b15b-abe28234f18c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e62f1-predictor-7d46c6569f-w87lj\" (UID: \"1161db69-91a5-43f1-b15b-abe28234f18c\") " pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:16.657941 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.657921 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1161db69-91a5-43f1-b15b-abe28234f18c-proxy-tls\") pod \"success-200-isvc-e62f1-predictor-7d46c6569f-w87lj\" (UID: \"1161db69-91a5-43f1-b15b-abe28234f18c\") " pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:16.666423 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.666401 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdgn\" (UniqueName: \"kubernetes.io/projected/1161db69-91a5-43f1-b15b-abe28234f18c-kube-api-access-7pdgn\") pod \"success-200-isvc-e62f1-predictor-7d46c6569f-w87lj\" (UID: \"1161db69-91a5-43f1-b15b-abe28234f18c\") " pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:16.771978 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.771943 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:16.850498 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.850446 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd"] Jul 16 17:45:16.853762 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.853741 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:16.855936 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.855913 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e62f1-predictor-serving-cert\"" Jul 16 17:45:16.855936 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.855930 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e62f1-kube-rbac-proxy-sar-config\"" Jul 16 17:45:16.865905 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.865878 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd"] Jul 16 17:45:16.908318 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.908291 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj"] Jul 16 17:45:16.911517 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:45:16.911446 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1161db69_91a5_43f1_b15b_abe28234f18c.slice/crio-0e8dc6534b649f4f372800809105fd42f2bb5c20f6bc68c6b1ea5c39e453a783 WatchSource:0}: Error finding container 0e8dc6534b649f4f372800809105fd42f2bb5c20f6bc68c6b1ea5c39e453a783: Status 404 returned error can't find the container with id 0e8dc6534b649f4f372800809105fd42f2bb5c20f6bc68c6b1ea5c39e453a783 Jul 16 17:45:16.940131 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.940103 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" event={"ID":"1161db69-91a5-43f1-b15b-abe28234f18c","Type":"ContainerStarted","Data":"0e8dc6534b649f4f372800809105fd42f2bb5c20f6bc68c6b1ea5c39e453a783"} Jul 16 17:45:16.957559 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.957538 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nv45\" (UniqueName: \"kubernetes.io/projected/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-kube-api-access-7nv45\") pod \"error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd\" (UID: \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\") " pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:16.957683 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.957577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-proxy-tls\") pod \"error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd\" (UID: \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\") " pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:16.957683 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:16.957616 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd\" (UID: \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\") " pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:17.059040 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:17.058959 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nv45\" (UniqueName: \"kubernetes.io/projected/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-kube-api-access-7nv45\") pod \"error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd\" (UID: \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\") " pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:17.059040 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:17.059003 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-proxy-tls\") pod \"error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd\" (UID: \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\") " pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:17.059040 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:17.059027 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd\" (UID: \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\") " pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:17.059333 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:45:17.059128 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-serving-cert: secret "error-404-isvc-e62f1-predictor-serving-cert" not found Jul 16 17:45:17.059333 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:45:17.059292 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-proxy-tls podName:c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524 nodeName:}" failed. No retries permitted until 2026-07-16 17:45:17.559192785 +0000 UTC m=+667.576422117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-proxy-tls") pod "error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" (UID: "c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524") : secret "error-404-isvc-e62f1-predictor-serving-cert" not found Jul 16 17:45:17.059782 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:17.059756 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd\" (UID: \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\") " pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:17.068100 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:17.068075 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nv45\" (UniqueName: \"kubernetes.io/projected/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-kube-api-access-7nv45\") pod \"error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd\" (UID: \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\") " pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:17.563564 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:17.563488 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-proxy-tls\") pod \"error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd\" (UID: \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\") " pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:17.566530 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:17.566503 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-proxy-tls\") pod \"error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd\" (UID: \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\") " pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:17.766828 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:17.766793 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:17.970193 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:17.970051 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd"] Jul 16 17:45:17.976538 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:45:17.976509 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ce7e22_0d4f_4cbc_82fb_d0a6096b0524.slice/crio-47c6d5be4af866aecf39e57bd54581c8c18eac4e7b95795ac808b20e446997dd WatchSource:0}: Error finding container 47c6d5be4af866aecf39e57bd54581c8c18eac4e7b95795ac808b20e446997dd: Status 404 returned error can't find the container with id 47c6d5be4af866aecf39e57bd54581c8c18eac4e7b95795ac808b20e446997dd Jul 16 17:45:18.958640 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:18.958600 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" event={"ID":"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524","Type":"ContainerStarted","Data":"47c6d5be4af866aecf39e57bd54581c8c18eac4e7b95795ac808b20e446997dd"} Jul 16 17:45:33.027993 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:33.027900 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" event={"ID":"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524","Type":"ContainerStarted","Data":"80d7f28cb2e2e972702718f206c6ccad8bd4cf95ee3b7745f0e177780a3e6628"} Jul 16 17:45:33.029434 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:33.029409 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" event={"ID":"1161db69-91a5-43f1-b15b-abe28234f18c","Type":"ContainerStarted","Data":"7c167fc45c9d11a12b49ebce509a43cb4a314a62dc45b6bc013dd569795e146b"} Jul 16 17:45:36.045438 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:36.045401 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" event={"ID":"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524","Type":"ContainerStarted","Data":"0cb9bcfd5808af50cf0624c934a6b2644d43a94afef332efc6551667b3417b53"} Jul 16 17:45:36.045925 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:36.045579 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:36.045925 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:36.045628 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:36.047110 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:36.047066 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Jul 16 17:45:36.047346 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:36.047323 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" event={"ID":"1161db69-91a5-43f1-b15b-abe28234f18c","Type":"ContainerStarted","Data":"2a8dfef7dc7552647970607289c79186594fe12ff95c887856e88e4fcb56ddf1"} Jul 16 17:45:36.047587 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:36.047571 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:36.047660 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:36.047614 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:36.048509 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:36.048488 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Jul 16 17:45:36.063677 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:36.063620 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" podStartSLOduration=2.818889161 podStartE2EDuration="20.063599724s" totalImagesPullingTime="17.244710563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:45:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:45:36.062370569 +0000 UTC m=+686.079599910" watchObservedRunningTime="2026-07-16 17:45:36.063599724 +0000 UTC m=+686.080829066" Jul 16 17:45:36.079678 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:36.079633 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" podStartSLOduration=1.775258467 podStartE2EDuration="20.07961893s" totalImagesPullingTime="18.304360463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:45:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:45:36.078027604 +0000 UTC m=+686.095256947" watchObservedRunningTime="2026-07-16 17:45:36.07961893 +0000 UTC m=+686.096848325" Jul 16 17:45:37.056047 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:37.056004 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Jul 16 17:45:37.056407 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:37.056220 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Jul 16 17:45:42.057933 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:42.057901 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:45:42.058515 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:42.058119 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:45:42.058515 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:42.058295 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Jul 16 17:45:42.058745 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:42.058721 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Jul 16 17:45:52.058597 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:52.058559 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Jul 16 17:45:52.059002 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:45:52.058704 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Jul 16 17:46:02.058955 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:02.058880 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Jul 16 17:46:02.059297 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:02.058877 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Jul 16 17:46:12.059004 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:12.058958 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused" Jul 16 17:46:12.059465 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:12.058958 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Jul 16 17:46:22.059224 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:22.059192 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:46:22.059845 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:22.059288 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:46:46.666240 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.666208 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd"] Jul 16 17:46:46.666686 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.666443 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kserve-container" containerID="cri-o://80d7f28cb2e2e972702718f206c6ccad8bd4cf95ee3b7745f0e177780a3e6628" gracePeriod=30 Jul 16 17:46:46.666686 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.666498 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kube-rbac-proxy" containerID="cri-o://0cb9bcfd5808af50cf0624c934a6b2644d43a94afef332efc6551667b3417b53" gracePeriod=30 Jul 16 17:46:46.713333 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.713305 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj"] Jul 16 17:46:46.713584 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.713562 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kserve-container" containerID="cri-o://7c167fc45c9d11a12b49ebce509a43cb4a314a62dc45b6bc013dd569795e146b" gracePeriod=30 Jul 16 17:46:46.713678 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.713573 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kube-rbac-proxy" containerID="cri-o://2a8dfef7dc7552647970607289c79186594fe12ff95c887856e88e4fcb56ddf1" gracePeriod=30 Jul 16 17:46:46.774200 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.774169 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94"] Jul 16 17:46:46.776511 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.776464 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:46.778775 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.778754 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f0d41-predictor-serving-cert\"" Jul 16 17:46:46.778850 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.778782 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f0d41-kube-rbac-proxy-sar-config\"" Jul 16 17:46:46.788537 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.788518 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94"] Jul 16 17:46:46.841951 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.841923 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/773df83f-1eb5-4d26-a88e-d6bd277ad72e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f0d41-predictor-8bd487845-z6k94\" (UID: \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\") " pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:46.842045 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.841968 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/773df83f-1eb5-4d26-a88e-d6bd277ad72e-proxy-tls\") pod \"success-200-isvc-f0d41-predictor-8bd487845-z6k94\" (UID: \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\") " pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:46.842045 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.842031 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbnqw\" (UniqueName: \"kubernetes.io/projected/773df83f-1eb5-4d26-a88e-d6bd277ad72e-kube-api-access-jbnqw\") pod \"success-200-isvc-f0d41-predictor-8bd487845-z6k94\" (UID: \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\") " pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:46.879870 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.879839 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d"] Jul 16 17:46:46.882651 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.882630 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:46.885069 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.885046 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f0d41-kube-rbac-proxy-sar-config\"" Jul 16 17:46:46.885169 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.885136 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f0d41-predictor-serving-cert\"" Jul 16 17:46:46.892043 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.892020 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d"] Jul 16 17:46:46.942872 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.942822 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/773df83f-1eb5-4d26-a88e-d6bd277ad72e-proxy-tls\") pod \"success-200-isvc-f0d41-predictor-8bd487845-z6k94\" (UID: \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\") " pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:46.942872 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.942860 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbnqw\" (UniqueName: \"kubernetes.io/projected/773df83f-1eb5-4d26-a88e-d6bd277ad72e-kube-api-access-jbnqw\") pod \"success-200-isvc-f0d41-predictor-8bd487845-z6k94\" (UID: \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\") " pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:46.942999 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.942904 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/773df83f-1eb5-4d26-a88e-d6bd277ad72e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f0d41-predictor-8bd487845-z6k94\" (UID: \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\") " pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:46.942999 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:46:46.942970 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-serving-cert: secret "success-200-isvc-f0d41-predictor-serving-cert" not found Jul 16 17:46:46.943081 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:46:46.943037 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/773df83f-1eb5-4d26-a88e-d6bd277ad72e-proxy-tls podName:773df83f-1eb5-4d26-a88e-d6bd277ad72e nodeName:}" failed. No retries permitted until 2026-07-16 17:46:47.443018559 +0000 UTC m=+757.460247880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/773df83f-1eb5-4d26-a88e-d6bd277ad72e-proxy-tls") pod "success-200-isvc-f0d41-predictor-8bd487845-z6k94" (UID: "773df83f-1eb5-4d26-a88e-d6bd277ad72e") : secret "success-200-isvc-f0d41-predictor-serving-cert" not found Jul 16 17:46:46.943528 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.943510 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/773df83f-1eb5-4d26-a88e-d6bd277ad72e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f0d41-predictor-8bd487845-z6k94\" (UID: \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\") " pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:46.954023 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:46.954002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbnqw\" (UniqueName: \"kubernetes.io/projected/773df83f-1eb5-4d26-a88e-d6bd277ad72e-kube-api-access-jbnqw\") pod \"success-200-isvc-f0d41-predictor-8bd487845-z6k94\" (UID: \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\") " pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:47.043814 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.043777 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2c8\" (UniqueName: \"kubernetes.io/projected/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-kube-api-access-jm2c8\") pod \"error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d\" (UID: \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:47.043964 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.043843 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d\" (UID: \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:47.043964 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.043869 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-proxy-tls\") pod \"error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d\" (UID: \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:47.053996 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.053960 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused" Jul 16 17:46:47.054104 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.053997 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused" Jul 16 17:46:47.144967 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.144937 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2c8\" (UniqueName: \"kubernetes.io/projected/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-kube-api-access-jm2c8\") pod \"error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d\" (UID: \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:47.145120 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.144991 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d\" (UID: \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:47.145120 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.145009 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-proxy-tls\") pod \"error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d\" (UID: \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:47.145120 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:46:47.145103 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-serving-cert: secret "error-404-isvc-f0d41-predictor-serving-cert" not found Jul 16 17:46:47.145280 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:46:47.145167 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-proxy-tls podName:b8f32b5c-c76d-40a4-aaf4-a07591e24eed nodeName:}" failed. No retries permitted until 2026-07-16 17:46:47.64515009 +0000 UTC m=+757.662379428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-proxy-tls") pod "error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" (UID: "b8f32b5c-c76d-40a4-aaf4-a07591e24eed") : secret "error-404-isvc-f0d41-predictor-serving-cert" not found Jul 16 17:46:47.145585 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.145567 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d\" (UID: \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:47.155658 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.155639 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2c8\" (UniqueName: \"kubernetes.io/projected/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-kube-api-access-jm2c8\") pod \"error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d\" (UID: \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:47.312648 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.312617 2582 generic.go:356] "Generic (PLEG): container finished" podID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerID="0cb9bcfd5808af50cf0624c934a6b2644d43a94afef332efc6551667b3417b53" exitCode=2 Jul 16 17:46:47.312803 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.312692 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" event={"ID":"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524","Type":"ContainerDied","Data":"0cb9bcfd5808af50cf0624c934a6b2644d43a94afef332efc6551667b3417b53"} Jul 16 17:46:47.314365 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.314339 2582 generic.go:356] "Generic (PLEG): container finished" podID="1161db69-91a5-43f1-b15b-abe28234f18c" containerID="2a8dfef7dc7552647970607289c79186594fe12ff95c887856e88e4fcb56ddf1" exitCode=2 Jul 16 17:46:47.314484 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.314374 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" event={"ID":"1161db69-91a5-43f1-b15b-abe28234f18c","Type":"ContainerDied","Data":"2a8dfef7dc7552647970607289c79186594fe12ff95c887856e88e4fcb56ddf1"} Jul 16 17:46:47.447384 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.447350 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/773df83f-1eb5-4d26-a88e-d6bd277ad72e-proxy-tls\") pod \"success-200-isvc-f0d41-predictor-8bd487845-z6k94\" (UID: \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\") " pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:47.449806 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.449784 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/773df83f-1eb5-4d26-a88e-d6bd277ad72e-proxy-tls\") pod \"success-200-isvc-f0d41-predictor-8bd487845-z6k94\" (UID: \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\") " pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:47.649141 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.649057 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-proxy-tls\") pod \"error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d\" (UID: \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:47.651538 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.651520 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-proxy-tls\") pod \"error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d\" (UID: \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:47.685525 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.685494 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:47.795107 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.795080 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:47.814780 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.814746 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94"] Jul 16 17:46:47.816989 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:46:47.816964 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773df83f_1eb5_4d26_a88e_d6bd277ad72e.slice/crio-a9a09a72d09033259c32b6f5ec8c2672b39ddb1b15df50e241a8817aace87d64 WatchSource:0}: Error finding container a9a09a72d09033259c32b6f5ec8c2672b39ddb1b15df50e241a8817aace87d64: Status 404 returned error can't find the container with id a9a09a72d09033259c32b6f5ec8c2672b39ddb1b15df50e241a8817aace87d64 Jul 16 17:46:47.933523 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:47.933496 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d"] Jul 16 17:46:47.935664 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:46:47.935623 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f32b5c_c76d_40a4_aaf4_a07591e24eed.slice/crio-ee62a54fbfbecd0cfdaede0da8e16985afc54c2f5e37c03bbd8e10cce6678dc1 WatchSource:0}: Error finding container ee62a54fbfbecd0cfdaede0da8e16985afc54c2f5e37c03bbd8e10cce6678dc1: Status 404 returned error can't find the container with id ee62a54fbfbecd0cfdaede0da8e16985afc54c2f5e37c03bbd8e10cce6678dc1 Jul 16 17:46:48.319482 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:48.319444 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" event={"ID":"b8f32b5c-c76d-40a4-aaf4-a07591e24eed","Type":"ContainerStarted","Data":"13395cafa425e20492ac4e2d916037df588821e46d5d6a2f108e31c65d3f5828"} Jul 16 17:46:48.319669 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:48.319541 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" event={"ID":"b8f32b5c-c76d-40a4-aaf4-a07591e24eed","Type":"ContainerStarted","Data":"948cfbe4f60fa65e355e21e32ebdf4971c764d986632747d18bfe67f05f64967"} Jul 16 17:46:48.319669 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:48.319576 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" event={"ID":"b8f32b5c-c76d-40a4-aaf4-a07591e24eed","Type":"ContainerStarted","Data":"ee62a54fbfbecd0cfdaede0da8e16985afc54c2f5e37c03bbd8e10cce6678dc1"} Jul 16 17:46:48.319669 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:48.319610 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:48.321229 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:48.321206 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" event={"ID":"773df83f-1eb5-4d26-a88e-d6bd277ad72e","Type":"ContainerStarted","Data":"898c017e5f1316e86e09b4d03021aecfeb7c582fcb0fb55429853babbc54768c"} Jul 16 17:46:48.321344 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:48.321256 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" event={"ID":"773df83f-1eb5-4d26-a88e-d6bd277ad72e","Type":"ContainerStarted","Data":"2244c4063f1e641a5ff34bb725a6fa4b5aa7d3cd73aaf6d8e4a87aaf0ae4ebbf"} Jul 16 17:46:48.321344 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:48.321279 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" event={"ID":"773df83f-1eb5-4d26-a88e-d6bd277ad72e","Type":"ContainerStarted","Data":"a9a09a72d09033259c32b6f5ec8c2672b39ddb1b15df50e241a8817aace87d64"} Jul 16 17:46:48.321344 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:48.321301 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:48.321454 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:48.321341 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:48.322463 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:48.322441 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Jul 16 17:46:48.338346 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:48.338304 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" podStartSLOduration=2.338292245 podStartE2EDuration="2.338292245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:46:48.337118183 +0000 UTC m=+758.354347550" watchObservedRunningTime="2026-07-16 17:46:48.338292245 +0000 UTC m=+758.355521587" Jul 16 17:46:48.355484 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:48.355428 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" podStartSLOduration=2.355415855 podStartE2EDuration="2.355415855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:46:48.354174919 +0000 UTC m=+758.371404263" watchObservedRunningTime="2026-07-16 17:46:48.355415855 +0000 UTC m=+758.372645189" Jul 16 17:46:49.325035 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:49.325002 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:49.325392 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:49.325043 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Jul 16 17:46:49.326339 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:49.326315 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Jul 16 17:46:49.913290 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:49.913266 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:46:50.066892 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.066858 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-proxy-tls\") pod \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\" (UID: \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\") " Jul 16 17:46:50.067023 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.066930 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-kube-rbac-proxy-sar-config\") pod \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\" (UID: \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\") " Jul 16 17:46:50.067023 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.066963 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nv45\" (UniqueName: \"kubernetes.io/projected/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-kube-api-access-7nv45\") pod \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\" (UID: \"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524\") " Jul 16 17:46:50.067299 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.067274 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" (UID: "c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:46:50.069242 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.069209 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-kube-api-access-7nv45" (OuterVolumeSpecName: "kube-api-access-7nv45") pod "c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" (UID: "c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524"). InnerVolumeSpecName "kube-api-access-7nv45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:46:50.069354 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.069316 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" (UID: "c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:46:50.141391 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.141369 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:46:50.167640 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.167620 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:46:50.167640 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.167641 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:46:50.167777 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.167652 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nv45\" (UniqueName: \"kubernetes.io/projected/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524-kube-api-access-7nv45\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:46:50.268831 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.268759 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pdgn\" (UniqueName: \"kubernetes.io/projected/1161db69-91a5-43f1-b15b-abe28234f18c-kube-api-access-7pdgn\") pod \"1161db69-91a5-43f1-b15b-abe28234f18c\" (UID: \"1161db69-91a5-43f1-b15b-abe28234f18c\") " Jul 16 17:46:50.268831 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.268792 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1161db69-91a5-43f1-b15b-abe28234f18c-kube-rbac-proxy-sar-config\") pod \"1161db69-91a5-43f1-b15b-abe28234f18c\" (UID: \"1161db69-91a5-43f1-b15b-abe28234f18c\") " Jul 16 17:46:50.268952 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.268834 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1161db69-91a5-43f1-b15b-abe28234f18c-proxy-tls\") pod \"1161db69-91a5-43f1-b15b-abe28234f18c\" (UID: \"1161db69-91a5-43f1-b15b-abe28234f18c\") " Jul 16 17:46:50.269148 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.269126 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1161db69-91a5-43f1-b15b-abe28234f18c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1161db69-91a5-43f1-b15b-abe28234f18c" (UID: "1161db69-91a5-43f1-b15b-abe28234f18c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:46:50.270938 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.270922 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1161db69-91a5-43f1-b15b-abe28234f18c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1161db69-91a5-43f1-b15b-abe28234f18c" (UID: "1161db69-91a5-43f1-b15b-abe28234f18c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:46:50.270987 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.270934 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1161db69-91a5-43f1-b15b-abe28234f18c-kube-api-access-7pdgn" (OuterVolumeSpecName: "kube-api-access-7pdgn") pod "1161db69-91a5-43f1-b15b-abe28234f18c" (UID: "1161db69-91a5-43f1-b15b-abe28234f18c"). InnerVolumeSpecName "kube-api-access-7pdgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:46:50.329699 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.329668 2582 generic.go:356] "Generic (PLEG): container finished" podID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerID="80d7f28cb2e2e972702718f206c6ccad8bd4cf95ee3b7745f0e177780a3e6628" exitCode=0 Jul 16 17:46:50.330041 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.329727 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" Jul 16 17:46:50.330041 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.329762 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" event={"ID":"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524","Type":"ContainerDied","Data":"80d7f28cb2e2e972702718f206c6ccad8bd4cf95ee3b7745f0e177780a3e6628"} Jul 16 17:46:50.330041 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.329823 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd" event={"ID":"c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524","Type":"ContainerDied","Data":"47c6d5be4af866aecf39e57bd54581c8c18eac4e7b95795ac808b20e446997dd"} Jul 16 17:46:50.330041 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.329858 2582 scope.go:117] "RemoveContainer" containerID="0cb9bcfd5808af50cf0624c934a6b2644d43a94afef332efc6551667b3417b53" Jul 16 17:46:50.331569 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.331542 2582 generic.go:356] "Generic (PLEG): container finished" podID="1161db69-91a5-43f1-b15b-abe28234f18c" containerID="7c167fc45c9d11a12b49ebce509a43cb4a314a62dc45b6bc013dd569795e146b" exitCode=0 Jul 16 17:46:50.331666 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.331577 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" Jul 16 17:46:50.331666 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.331606 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" event={"ID":"1161db69-91a5-43f1-b15b-abe28234f18c","Type":"ContainerDied","Data":"7c167fc45c9d11a12b49ebce509a43cb4a314a62dc45b6bc013dd569795e146b"} Jul 16 17:46:50.331666 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.331660 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj" event={"ID":"1161db69-91a5-43f1-b15b-abe28234f18c","Type":"ContainerDied","Data":"0e8dc6534b649f4f372800809105fd42f2bb5c20f6bc68c6b1ea5c39e453a783"} Jul 16 17:46:50.332052 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.332024 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Jul 16 17:46:50.338872 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.338853 2582 scope.go:117] "RemoveContainer" containerID="80d7f28cb2e2e972702718f206c6ccad8bd4cf95ee3b7745f0e177780a3e6628" Jul 16 17:46:50.346247 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.346232 2582 scope.go:117] "RemoveContainer" containerID="0cb9bcfd5808af50cf0624c934a6b2644d43a94afef332efc6551667b3417b53" Jul 16 17:46:50.346549 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:46:50.346458 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb9bcfd5808af50cf0624c934a6b2644d43a94afef332efc6551667b3417b53\": container with ID starting with 0cb9bcfd5808af50cf0624c934a6b2644d43a94afef332efc6551667b3417b53 not found: ID does not exist" containerID="0cb9bcfd5808af50cf0624c934a6b2644d43a94afef332efc6551667b3417b53" Jul 16 17:46:50.346549 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.346511 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb9bcfd5808af50cf0624c934a6b2644d43a94afef332efc6551667b3417b53"} err="failed to get container status \"0cb9bcfd5808af50cf0624c934a6b2644d43a94afef332efc6551667b3417b53\": rpc error: code = NotFound desc = could not find container \"0cb9bcfd5808af50cf0624c934a6b2644d43a94afef332efc6551667b3417b53\": container with ID starting with 0cb9bcfd5808af50cf0624c934a6b2644d43a94afef332efc6551667b3417b53 not found: ID does not exist" Jul 16 17:46:50.346549 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.346536 2582 scope.go:117] "RemoveContainer" containerID="80d7f28cb2e2e972702718f206c6ccad8bd4cf95ee3b7745f0e177780a3e6628" Jul 16 17:46:50.346767 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:46:50.346742 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d7f28cb2e2e972702718f206c6ccad8bd4cf95ee3b7745f0e177780a3e6628\": container with ID starting with 80d7f28cb2e2e972702718f206c6ccad8bd4cf95ee3b7745f0e177780a3e6628 not found: ID does not exist" containerID="80d7f28cb2e2e972702718f206c6ccad8bd4cf95ee3b7745f0e177780a3e6628" Jul 16 17:46:50.346806 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.346778 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d7f28cb2e2e972702718f206c6ccad8bd4cf95ee3b7745f0e177780a3e6628"} err="failed to get container status \"80d7f28cb2e2e972702718f206c6ccad8bd4cf95ee3b7745f0e177780a3e6628\": rpc error: code = NotFound desc = could not find container \"80d7f28cb2e2e972702718f206c6ccad8bd4cf95ee3b7745f0e177780a3e6628\": container with ID starting with 80d7f28cb2e2e972702718f206c6ccad8bd4cf95ee3b7745f0e177780a3e6628 not found: ID does not exist" Jul 16 17:46:50.346806 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.346799 2582 scope.go:117] "RemoveContainer" containerID="2a8dfef7dc7552647970607289c79186594fe12ff95c887856e88e4fcb56ddf1" Jul 16 17:46:50.353579 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.353551 2582 scope.go:117] "RemoveContainer" containerID="7c167fc45c9d11a12b49ebce509a43cb4a314a62dc45b6bc013dd569795e146b" Jul 16 17:46:50.355675 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.355655 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj"] Jul 16 17:46:50.360381 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.360356 2582 scope.go:117] "RemoveContainer" containerID="2a8dfef7dc7552647970607289c79186594fe12ff95c887856e88e4fcb56ddf1" Jul 16 17:46:50.360674 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:46:50.360658 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8dfef7dc7552647970607289c79186594fe12ff95c887856e88e4fcb56ddf1\": container with ID starting with 2a8dfef7dc7552647970607289c79186594fe12ff95c887856e88e4fcb56ddf1 not found: ID does not exist" containerID="2a8dfef7dc7552647970607289c79186594fe12ff95c887856e88e4fcb56ddf1" Jul 16 17:46:50.360728 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.360682 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8dfef7dc7552647970607289c79186594fe12ff95c887856e88e4fcb56ddf1"} err="failed to get container status \"2a8dfef7dc7552647970607289c79186594fe12ff95c887856e88e4fcb56ddf1\": rpc error: code = NotFound desc = could not find container \"2a8dfef7dc7552647970607289c79186594fe12ff95c887856e88e4fcb56ddf1\": container with ID starting with 2a8dfef7dc7552647970607289c79186594fe12ff95c887856e88e4fcb56ddf1 not found: ID does not exist" Jul 16 17:46:50.360728 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.360699 2582 scope.go:117] "RemoveContainer" containerID="7c167fc45c9d11a12b49ebce509a43cb4a314a62dc45b6bc013dd569795e146b" Jul 16 17:46:50.360955 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:46:50.360940 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c167fc45c9d11a12b49ebce509a43cb4a314a62dc45b6bc013dd569795e146b\": container with ID starting with 7c167fc45c9d11a12b49ebce509a43cb4a314a62dc45b6bc013dd569795e146b not found: ID does not exist" containerID="7c167fc45c9d11a12b49ebce509a43cb4a314a62dc45b6bc013dd569795e146b" Jul 16 17:46:50.361007 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.360983 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c167fc45c9d11a12b49ebce509a43cb4a314a62dc45b6bc013dd569795e146b"} err="failed to get container status \"7c167fc45c9d11a12b49ebce509a43cb4a314a62dc45b6bc013dd569795e146b\": rpc error: code = NotFound desc = could not find container \"7c167fc45c9d11a12b49ebce509a43cb4a314a62dc45b6bc013dd569795e146b\": container with ID starting with 7c167fc45c9d11a12b49ebce509a43cb4a314a62dc45b6bc013dd569795e146b not found: ID does not exist" Jul 16 17:46:50.363301 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.363281 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e62f1-predictor-7d46c6569f-w87lj"] Jul 16 17:46:50.369721 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.369704 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pdgn\" (UniqueName: \"kubernetes.io/projected/1161db69-91a5-43f1-b15b-abe28234f18c-kube-api-access-7pdgn\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:46:50.369776 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.369723 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1161db69-91a5-43f1-b15b-abe28234f18c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:46:50.369776 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.369734 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1161db69-91a5-43f1-b15b-abe28234f18c-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:46:50.375357 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.375337 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd"] Jul 16 17:46:50.386977 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.386955 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e62f1-predictor-57b4c448cd-lhtxd"] Jul 16 17:46:50.584717 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.584651 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" path="/var/lib/kubelet/pods/1161db69-91a5-43f1-b15b-abe28234f18c/volumes" Jul 16 17:46:50.585066 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:50.585052 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" path="/var/lib/kubelet/pods/c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524/volumes" Jul 16 17:46:54.329738 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:54.329710 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:46:54.330323 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:54.330297 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Jul 16 17:46:55.336277 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:55.336244 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:46:55.336661 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:46:55.336637 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Jul 16 17:47:04.330542 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:04.330502 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Jul 16 17:47:05.336848 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:05.336805 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Jul 16 17:47:14.330970 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:14.330919 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Jul 16 17:47:15.337613 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:15.337571 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Jul 16 17:47:24.330409 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:24.330370 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Jul 16 17:47:25.336717 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:25.336678 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Jul 16 17:47:26.817378 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.817345 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h"] Jul 16 17:47:26.817915 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.817788 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kserve-container" Jul 16 17:47:26.817915 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.817806 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kserve-container" Jul 16 17:47:26.817915 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.817832 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kube-rbac-proxy" Jul 16 17:47:26.817915 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.817841 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kube-rbac-proxy" Jul 16 17:47:26.817915 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.817859 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kserve-container" Jul 16 17:47:26.817915 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.817867 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kserve-container" Jul 16 17:47:26.817915 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.817878 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kube-rbac-proxy" Jul 16 17:47:26.817915 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.817886 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kube-rbac-proxy" Jul 16 17:47:26.818297 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.817973 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kserve-container" Jul 16 17:47:26.818297 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.817984 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kube-rbac-proxy" Jul 16 17:47:26.818297 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.817993 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1161db69-91a5-43f1-b15b-abe28234f18c" containerName="kserve-container" Jul 16 17:47:26.818297 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.818006 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6ce7e22-0d4f-4cbc-82fb-d0a6096b0524" containerName="kube-rbac-proxy" Jul 16 17:47:26.820956 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.820937 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:26.823254 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.823233 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-38e88-predictor-serving-cert\"" Jul 16 17:47:26.824137 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.824118 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-38e88-kube-rbac-proxy-sar-config\"" Jul 16 17:47:26.837094 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.837062 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h"] Jul 16 17:47:26.843287 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.843262 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536a3642-b49f-40f3-97f8-9c325a55a02d-proxy-tls\") pod \"success-200-isvc-38e88-predictor-5d744d6567-8cj6h\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:26.843426 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.843299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb2h9\" (UniqueName: \"kubernetes.io/projected/536a3642-b49f-40f3-97f8-9c325a55a02d-kube-api-access-rb2h9\") pod \"success-200-isvc-38e88-predictor-5d744d6567-8cj6h\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:26.843426 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.843318 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536a3642-b49f-40f3-97f8-9c325a55a02d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-38e88-predictor-5d744d6567-8cj6h\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:26.935195 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.935163 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds"] Jul 16 17:47:26.940914 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.939149 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:26.942708 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.942684 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-38e88-predictor-serving-cert\"" Jul 16 17:47:26.942832 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.942756 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-38e88-kube-rbac-proxy-sar-config\"" Jul 16 17:47:26.944340 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.944318 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb2h9\" (UniqueName: \"kubernetes.io/projected/536a3642-b49f-40f3-97f8-9c325a55a02d-kube-api-access-rb2h9\") pod \"success-200-isvc-38e88-predictor-5d744d6567-8cj6h\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:26.944422 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.944368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536a3642-b49f-40f3-97f8-9c325a55a02d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-38e88-predictor-5d744d6567-8cj6h\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:26.944525 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.944508 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536a3642-b49f-40f3-97f8-9c325a55a02d-proxy-tls\") pod \"success-200-isvc-38e88-predictor-5d744d6567-8cj6h\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:26.944623 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:47:26.944609 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-38e88-predictor-serving-cert: secret "success-200-isvc-38e88-predictor-serving-cert" not found Jul 16 17:47:26.944687 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:47:26.944663 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/536a3642-b49f-40f3-97f8-9c325a55a02d-proxy-tls podName:536a3642-b49f-40f3-97f8-9c325a55a02d nodeName:}" failed. No retries permitted until 2026-07-16 17:47:27.444650228 +0000 UTC m=+797.461879548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/536a3642-b49f-40f3-97f8-9c325a55a02d-proxy-tls") pod "success-200-isvc-38e88-predictor-5d744d6567-8cj6h" (UID: "536a3642-b49f-40f3-97f8-9c325a55a02d") : secret "success-200-isvc-38e88-predictor-serving-cert" not found Jul 16 17:47:26.945040 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.945019 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536a3642-b49f-40f3-97f8-9c325a55a02d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-38e88-predictor-5d744d6567-8cj6h\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:26.949157 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.949133 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds"] Jul 16 17:47:26.956408 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:26.956386 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb2h9\" (UniqueName: \"kubernetes.io/projected/536a3642-b49f-40f3-97f8-9c325a55a02d-kube-api-access-rb2h9\") pod \"success-200-isvc-38e88-predictor-5d744d6567-8cj6h\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:27.045541 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.045506 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32b3e31a-3cea-433f-84c1-6d4da383701b-proxy-tls\") pod \"error-404-isvc-38e88-predictor-859fd4f7c8-bfsds\" (UID: \"32b3e31a-3cea-433f-84c1-6d4da383701b\") " pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:27.045708 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.045547 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sgqv\" (UniqueName: \"kubernetes.io/projected/32b3e31a-3cea-433f-84c1-6d4da383701b-kube-api-access-2sgqv\") pod \"error-404-isvc-38e88-predictor-859fd4f7c8-bfsds\" (UID: \"32b3e31a-3cea-433f-84c1-6d4da383701b\") " pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:27.045708 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.045590 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32b3e31a-3cea-433f-84c1-6d4da383701b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-38e88-predictor-859fd4f7c8-bfsds\" (UID: \"32b3e31a-3cea-433f-84c1-6d4da383701b\") " pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:27.146507 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.146406 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32b3e31a-3cea-433f-84c1-6d4da383701b-proxy-tls\") pod \"error-404-isvc-38e88-predictor-859fd4f7c8-bfsds\" (UID: \"32b3e31a-3cea-433f-84c1-6d4da383701b\") " pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:27.146507 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.146454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sgqv\" (UniqueName: \"kubernetes.io/projected/32b3e31a-3cea-433f-84c1-6d4da383701b-kube-api-access-2sgqv\") pod \"error-404-isvc-38e88-predictor-859fd4f7c8-bfsds\" (UID: \"32b3e31a-3cea-433f-84c1-6d4da383701b\") " pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:27.146697 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.146537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32b3e31a-3cea-433f-84c1-6d4da383701b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-38e88-predictor-859fd4f7c8-bfsds\" (UID: \"32b3e31a-3cea-433f-84c1-6d4da383701b\") " pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:27.147215 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.147150 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32b3e31a-3cea-433f-84c1-6d4da383701b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-38e88-predictor-859fd4f7c8-bfsds\" (UID: \"32b3e31a-3cea-433f-84c1-6d4da383701b\") " pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:27.149068 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.149041 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32b3e31a-3cea-433f-84c1-6d4da383701b-proxy-tls\") pod \"error-404-isvc-38e88-predictor-859fd4f7c8-bfsds\" (UID: \"32b3e31a-3cea-433f-84c1-6d4da383701b\") " pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:27.156090 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.156068 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sgqv\" (UniqueName: \"kubernetes.io/projected/32b3e31a-3cea-433f-84c1-6d4da383701b-kube-api-access-2sgqv\") pod \"error-404-isvc-38e88-predictor-859fd4f7c8-bfsds\" (UID: \"32b3e31a-3cea-433f-84c1-6d4da383701b\") " pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:27.251392 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.251342 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:27.383745 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.383715 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds"] Jul 16 17:47:27.385852 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:47:27.385824 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b3e31a_3cea_433f_84c1_6d4da383701b.slice/crio-76d338f70a7f0c87c1af7e82e9299eec2fb4f26e145a80c39a27b5b12029ad8e WatchSource:0}: Error finding container 76d338f70a7f0c87c1af7e82e9299eec2fb4f26e145a80c39a27b5b12029ad8e: Status 404 returned error can't find the container with id 76d338f70a7f0c87c1af7e82e9299eec2fb4f26e145a80c39a27b5b12029ad8e Jul 16 17:47:27.449045 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.449027 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536a3642-b49f-40f3-97f8-9c325a55a02d-proxy-tls\") pod \"success-200-isvc-38e88-predictor-5d744d6567-8cj6h\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:27.449150 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:47:27.449137 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-38e88-predictor-serving-cert: secret "success-200-isvc-38e88-predictor-serving-cert" not found Jul 16 17:47:27.449193 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:47:27.449185 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/536a3642-b49f-40f3-97f8-9c325a55a02d-proxy-tls podName:536a3642-b49f-40f3-97f8-9c325a55a02d nodeName:}" failed. No retries permitted until 2026-07-16 17:47:28.449171537 +0000 UTC m=+798.466400857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/536a3642-b49f-40f3-97f8-9c325a55a02d-proxy-tls") pod "success-200-isvc-38e88-predictor-5d744d6567-8cj6h" (UID: "536a3642-b49f-40f3-97f8-9c325a55a02d") : secret "success-200-isvc-38e88-predictor-serving-cert" not found Jul 16 17:47:27.466359 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.466334 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" event={"ID":"32b3e31a-3cea-433f-84c1-6d4da383701b","Type":"ContainerStarted","Data":"ba7098c56428e18c52238ddf0982139c8378b0a73a307883d6fb89b0553ea384"} Jul 16 17:47:27.466443 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:27.466384 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" event={"ID":"32b3e31a-3cea-433f-84c1-6d4da383701b","Type":"ContainerStarted","Data":"76d338f70a7f0c87c1af7e82e9299eec2fb4f26e145a80c39a27b5b12029ad8e"} Jul 16 17:47:28.459866 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:28.459826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536a3642-b49f-40f3-97f8-9c325a55a02d-proxy-tls\") pod \"success-200-isvc-38e88-predictor-5d744d6567-8cj6h\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:28.462442 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:28.462409 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536a3642-b49f-40f3-97f8-9c325a55a02d-proxy-tls\") pod \"success-200-isvc-38e88-predictor-5d744d6567-8cj6h\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:28.472838 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:28.472800 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" event={"ID":"32b3e31a-3cea-433f-84c1-6d4da383701b","Type":"ContainerStarted","Data":"cdf66a32cfdf7c75b32583ff00a3d22b3235748ddbbbe7a5ad160ae0ff72464d"} Jul 16 17:47:28.472986 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:28.472971 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:28.492014 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:28.491963 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" podStartSLOduration=2.491949932 podStartE2EDuration="2.491949932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:47:28.489806172 +0000 UTC m=+798.507035514" watchObservedRunningTime="2026-07-16 17:47:28.491949932 +0000 UTC m=+798.509179330" Jul 16 17:47:28.630362 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:28.630329 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:28.756321 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:28.756295 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h"] Jul 16 17:47:28.758727 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:47:28.758698 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536a3642_b49f_40f3_97f8_9c325a55a02d.slice/crio-a53900cee7a8a61454e94e98055227371006b398d8468f3e66b9254dc19eca4a WatchSource:0}: Error finding container a53900cee7a8a61454e94e98055227371006b398d8468f3e66b9254dc19eca4a: Status 404 returned error can't find the container with id a53900cee7a8a61454e94e98055227371006b398d8468f3e66b9254dc19eca4a Jul 16 17:47:29.478357 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:29.478314 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" event={"ID":"536a3642-b49f-40f3-97f8-9c325a55a02d","Type":"ContainerStarted","Data":"bc8d901a08ba16ed3ee2876249da7c0a14283e0a79e04bf11139779a2dd40fdc"} Jul 16 17:47:29.479246 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:29.478396 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" event={"ID":"536a3642-b49f-40f3-97f8-9c325a55a02d","Type":"ContainerStarted","Data":"726202d58c026062b90f00a2ec24cf1b09bf568df87adb09d9e2e61048cf6af6"} Jul 16 17:47:29.479246 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:29.478430 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" event={"ID":"536a3642-b49f-40f3-97f8-9c325a55a02d","Type":"ContainerStarted","Data":"a53900cee7a8a61454e94e98055227371006b398d8468f3e66b9254dc19eca4a"} Jul 16 17:47:29.479246 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:29.478834 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:29.479246 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:29.478889 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:29.480115 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:29.480081 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 16 17:47:29.500031 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:29.499985 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" podStartSLOduration=3.499971102 podStartE2EDuration="3.499971102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:47:29.49815072 +0000 UTC m=+799.515380062" watchObservedRunningTime="2026-07-16 17:47:29.499971102 +0000 UTC m=+799.517200455" Jul 16 17:47:30.485399 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:30.485365 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:30.485996 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:30.485639 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 16 17:47:30.486814 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:30.486786 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 16 17:47:31.489261 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:31.489221 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 16 17:47:34.331046 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:34.331016 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:47:35.337632 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:35.337601 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:47:35.490073 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:35.490048 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:47:35.490455 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:35.490428 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 16 17:47:36.494163 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:36.494133 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:47:36.494662 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:36.494636 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 16 17:47:45.490893 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:45.490851 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 16 17:47:46.494890 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:46.494851 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 16 17:47:55.491453 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:55.491409 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 16 17:47:56.494843 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:47:56.494805 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 16 17:48:05.490603 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:48:05.490561 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 16 17:48:06.495125 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:48:06.495090 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 16 17:48:15.491654 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:48:15.491626 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:48:16.495829 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:48:16.495801 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:49:10.435967 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:49:10.435897 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 17:49:10.437520 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:49:10.437498 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 17:49:10.442175 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:49:10.442153 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 17:49:10.443048 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:49:10.443033 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 17:54:10.459901 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:54:10.459828 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 17:54:10.462675 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:54:10.462654 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 17:54:10.465750 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:54:10.465728 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 17:54:10.468341 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:54:10.468323 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 17:56:01.761379 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.761346 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94"] Jul 16 17:56:01.763911 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.761610 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kserve-container" containerID="cri-o://2244c4063f1e641a5ff34bb725a6fa4b5aa7d3cd73aaf6d8e4a87aaf0ae4ebbf" gracePeriod=30 Jul 16 17:56:01.763911 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.761649 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kube-rbac-proxy" containerID="cri-o://898c017e5f1316e86e09b4d03021aecfeb7c582fcb0fb55429853babbc54768c" gracePeriod=30 Jul 16 17:56:01.867066 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.867034 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj"] Jul 16 17:56:01.870463 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.870444 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:01.873059 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.873034 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1da67-predictor-serving-cert\"" Jul 16 17:56:01.873059 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.873039 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1da67-kube-rbac-proxy-sar-config\"" Jul 16 17:56:01.881506 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.881485 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj"] Jul 16 17:56:01.888854 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.888830 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d"] Jul 16 17:56:01.889037 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.889018 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kserve-container" containerID="cri-o://948cfbe4f60fa65e355e21e32ebdf4971c764d986632747d18bfe67f05f64967" gracePeriod=30 Jul 16 17:56:01.889115 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.889094 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kube-rbac-proxy" containerID="cri-o://13395cafa425e20492ac4e2d916037df588821e46d5d6a2f108e31c65d3f5828" gracePeriod=30 Jul 16 17:56:01.937389 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.937361 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgzj5\" (UniqueName: \"kubernetes.io/projected/d9452ec7-5072-4e69-ab92-fd9c40131e0a-kube-api-access-kgzj5\") pod \"success-200-isvc-1da67-predictor-76bff85dd9-db9wj\" (UID: \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\") " pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:01.937513 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.937396 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9452ec7-5072-4e69-ab92-fd9c40131e0a-proxy-tls\") pod \"success-200-isvc-1da67-predictor-76bff85dd9-db9wj\" (UID: \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\") " pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:01.937513 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.937415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9452ec7-5072-4e69-ab92-fd9c40131e0a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1da67-predictor-76bff85dd9-db9wj\" (UID: \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\") " pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:01.966210 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.966184 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr"] Jul 16 17:56:01.969547 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.969531 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:01.971910 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.971885 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1da67-predictor-serving-cert\"" Jul 16 17:56:01.972022 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.971885 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1da67-kube-rbac-proxy-sar-config\"" Jul 16 17:56:01.979838 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:01.979819 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr"] Jul 16 17:56:02.037994 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.037920 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgqx8\" (UniqueName: \"kubernetes.io/projected/b1e18acf-38af-4f3e-87db-3375d62803a9-kube-api-access-pgqx8\") pod \"error-404-isvc-1da67-predictor-5c69b7464d-wgnzr\" (UID: \"b1e18acf-38af-4f3e-87db-3375d62803a9\") " pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:02.038097 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.038043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgzj5\" (UniqueName: \"kubernetes.io/projected/d9452ec7-5072-4e69-ab92-fd9c40131e0a-kube-api-access-kgzj5\") pod \"success-200-isvc-1da67-predictor-76bff85dd9-db9wj\" (UID: \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\") " pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:02.038097 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.038080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1e18acf-38af-4f3e-87db-3375d62803a9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1da67-predictor-5c69b7464d-wgnzr\" (UID: \"b1e18acf-38af-4f3e-87db-3375d62803a9\") " pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:02.038203 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.038112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9452ec7-5072-4e69-ab92-fd9c40131e0a-proxy-tls\") pod \"success-200-isvc-1da67-predictor-76bff85dd9-db9wj\" (UID: \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\") " pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:02.038203 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.038140 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9452ec7-5072-4e69-ab92-fd9c40131e0a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1da67-predictor-76bff85dd9-db9wj\" (UID: \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\") " pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:02.038203 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.038167 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e18acf-38af-4f3e-87db-3375d62803a9-proxy-tls\") pod \"error-404-isvc-1da67-predictor-5c69b7464d-wgnzr\" (UID: \"b1e18acf-38af-4f3e-87db-3375d62803a9\") " pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:02.038360 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:56:02.038235 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-1da67-predictor-serving-cert: secret "success-200-isvc-1da67-predictor-serving-cert" not found Jul 16 17:56:02.038360 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:56:02.038294 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9452ec7-5072-4e69-ab92-fd9c40131e0a-proxy-tls podName:d9452ec7-5072-4e69-ab92-fd9c40131e0a nodeName:}" failed. No retries permitted until 2026-07-16 17:56:02.538274414 +0000 UTC m=+1312.555503735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d9452ec7-5072-4e69-ab92-fd9c40131e0a-proxy-tls") pod "success-200-isvc-1da67-predictor-76bff85dd9-db9wj" (UID: "d9452ec7-5072-4e69-ab92-fd9c40131e0a") : secret "success-200-isvc-1da67-predictor-serving-cert" not found Jul 16 17:56:02.038836 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.038815 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9452ec7-5072-4e69-ab92-fd9c40131e0a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1da67-predictor-76bff85dd9-db9wj\" (UID: \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\") " pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:02.046449 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.046430 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgzj5\" (UniqueName: \"kubernetes.io/projected/d9452ec7-5072-4e69-ab92-fd9c40131e0a-kube-api-access-kgzj5\") pod \"success-200-isvc-1da67-predictor-76bff85dd9-db9wj\" (UID: \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\") " pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:02.139175 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.139147 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgqx8\" (UniqueName: \"kubernetes.io/projected/b1e18acf-38af-4f3e-87db-3375d62803a9-kube-api-access-pgqx8\") pod \"error-404-isvc-1da67-predictor-5c69b7464d-wgnzr\" (UID: \"b1e18acf-38af-4f3e-87db-3375d62803a9\") " pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:02.139279 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.139205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1e18acf-38af-4f3e-87db-3375d62803a9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1da67-predictor-5c69b7464d-wgnzr\" (UID: \"b1e18acf-38af-4f3e-87db-3375d62803a9\") " pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:02.139453 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.139430 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e18acf-38af-4f3e-87db-3375d62803a9-proxy-tls\") pod \"error-404-isvc-1da67-predictor-5c69b7464d-wgnzr\" (UID: \"b1e18acf-38af-4f3e-87db-3375d62803a9\") " pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:02.139588 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:56:02.139570 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1da67-predictor-serving-cert: secret "error-404-isvc-1da67-predictor-serving-cert" not found Jul 16 17:56:02.139649 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:56:02.139641 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1e18acf-38af-4f3e-87db-3375d62803a9-proxy-tls podName:b1e18acf-38af-4f3e-87db-3375d62803a9 nodeName:}" failed. No retries permitted until 2026-07-16 17:56:02.639623531 +0000 UTC m=+1312.656852869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b1e18acf-38af-4f3e-87db-3375d62803a9-proxy-tls") pod "error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" (UID: "b1e18acf-38af-4f3e-87db-3375d62803a9") : secret "error-404-isvc-1da67-predictor-serving-cert" not found Jul 16 17:56:02.139712 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.139699 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1e18acf-38af-4f3e-87db-3375d62803a9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1da67-predictor-5c69b7464d-wgnzr\" (UID: \"b1e18acf-38af-4f3e-87db-3375d62803a9\") " pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:02.147621 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.147601 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgqx8\" (UniqueName: \"kubernetes.io/projected/b1e18acf-38af-4f3e-87db-3375d62803a9-kube-api-access-pgqx8\") pod \"error-404-isvc-1da67-predictor-5c69b7464d-wgnzr\" (UID: \"b1e18acf-38af-4f3e-87db-3375d62803a9\") " pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:02.441626 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.441537 2582 generic.go:356] "Generic (PLEG): container finished" podID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerID="13395cafa425e20492ac4e2d916037df588821e46d5d6a2f108e31c65d3f5828" exitCode=2 Jul 16 17:56:02.441764 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.441616 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" event={"ID":"b8f32b5c-c76d-40a4-aaf4-a07591e24eed","Type":"ContainerDied","Data":"13395cafa425e20492ac4e2d916037df588821e46d5d6a2f108e31c65d3f5828"} Jul 16 17:56:02.443231 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.443210 2582 generic.go:356] "Generic (PLEG): container finished" podID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerID="898c017e5f1316e86e09b4d03021aecfeb7c582fcb0fb55429853babbc54768c" exitCode=2 Jul 16 17:56:02.443344 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.443246 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" event={"ID":"773df83f-1eb5-4d26-a88e-d6bd277ad72e","Type":"ContainerDied","Data":"898c017e5f1316e86e09b4d03021aecfeb7c582fcb0fb55429853babbc54768c"} Jul 16 17:56:02.542921 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.542886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9452ec7-5072-4e69-ab92-fd9c40131e0a-proxy-tls\") pod \"success-200-isvc-1da67-predictor-76bff85dd9-db9wj\" (UID: \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\") " pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:02.545350 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.545327 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9452ec7-5072-4e69-ab92-fd9c40131e0a-proxy-tls\") pod \"success-200-isvc-1da67-predictor-76bff85dd9-db9wj\" (UID: \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\") " pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:02.644177 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.644149 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e18acf-38af-4f3e-87db-3375d62803a9-proxy-tls\") pod \"error-404-isvc-1da67-predictor-5c69b7464d-wgnzr\" (UID: \"b1e18acf-38af-4f3e-87db-3375d62803a9\") " pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:02.646504 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.646486 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e18acf-38af-4f3e-87db-3375d62803a9-proxy-tls\") pod \"error-404-isvc-1da67-predictor-5c69b7464d-wgnzr\" (UID: \"b1e18acf-38af-4f3e-87db-3375d62803a9\") " pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:02.782197 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.782171 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:02.880829 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.880796 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:02.917602 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:02.917576 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj"] Jul 16 17:56:02.919423 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:56:02.919395 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9452ec7_5072_4e69_ab92_fd9c40131e0a.slice/crio-0db49cc7ae2e40bbfdf0e66f97a17d7ba574554c241022612e30bddde3bd3eab WatchSource:0}: Error finding container 0db49cc7ae2e40bbfdf0e66f97a17d7ba574554c241022612e30bddde3bd3eab: Status 404 returned error can't find the container with id 0db49cc7ae2e40bbfdf0e66f97a17d7ba574554c241022612e30bddde3bd3eab Jul 16 17:56:03.023546 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.023523 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr"] Jul 16 17:56:03.031431 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:56:03.031409 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e18acf_38af_4f3e_87db_3375d62803a9.slice/crio-00168229cd58cbc47328ecbdae5fc94574c8c2ab6d44bacbf7d66b9c03a7be92 WatchSource:0}: Error finding container 00168229cd58cbc47328ecbdae5fc94574c8c2ab6d44bacbf7d66b9c03a7be92: Status 404 returned error can't find the container with id 00168229cd58cbc47328ecbdae5fc94574c8c2ab6d44bacbf7d66b9c03a7be92 Jul 16 17:56:03.449279 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.449172 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" event={"ID":"b1e18acf-38af-4f3e-87db-3375d62803a9","Type":"ContainerStarted","Data":"e663bfb99eff297649c05bfe88696be0ab993ca5ff3335e24904c9b05e8b3288"} Jul 16 17:56:03.449279 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.449257 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:03.449532 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.449290 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" event={"ID":"b1e18acf-38af-4f3e-87db-3375d62803a9","Type":"ContainerStarted","Data":"3b0c903fbd29ecb96756ee86f7bc85ed36dc1fa634a376fc2c7cef97fdc8dbb8"} Jul 16 17:56:03.449532 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.449314 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:03.449532 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.449341 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" event={"ID":"b1e18acf-38af-4f3e-87db-3375d62803a9","Type":"ContainerStarted","Data":"00168229cd58cbc47328ecbdae5fc94574c8c2ab6d44bacbf7d66b9c03a7be92"} Jul 16 17:56:03.450613 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.450578 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Jul 16 17:56:03.451118 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.451097 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" event={"ID":"d9452ec7-5072-4e69-ab92-fd9c40131e0a","Type":"ContainerStarted","Data":"edff043dbc1175b8269ef905b826f5a5721a36728f6332fccd584dfbc30e3bb0"} Jul 16 17:56:03.451209 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.451146 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" event={"ID":"d9452ec7-5072-4e69-ab92-fd9c40131e0a","Type":"ContainerStarted","Data":"3042e174bff378095f4cb9a83c70dbdf86b69b9af6504635e64a0fb63a97f724"} Jul 16 17:56:03.451209 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.451185 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:03.451282 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.451211 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" event={"ID":"d9452ec7-5072-4e69-ab92-fd9c40131e0a","Type":"ContainerStarted","Data":"0db49cc7ae2e40bbfdf0e66f97a17d7ba574554c241022612e30bddde3bd3eab"} Jul 16 17:56:03.451282 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.451231 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:03.452190 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.452168 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 16 17:56:03.466942 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.466904 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" podStartSLOduration=2.466889783 podStartE2EDuration="2.466889783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:56:03.46537236 +0000 UTC m=+1313.482601712" watchObservedRunningTime="2026-07-16 17:56:03.466889783 +0000 UTC m=+1313.484119117" Jul 16 17:56:03.485242 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:03.485190 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" podStartSLOduration=2.485177539 podStartE2EDuration="2.485177539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:56:03.483044325 +0000 UTC m=+1313.500273686" watchObservedRunningTime="2026-07-16 17:56:03.485177539 +0000 UTC m=+1313.502406881" Jul 16 17:56:04.326048 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:04.325996 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused" Jul 16 17:56:04.330340 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:04.330316 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Jul 16 17:56:04.455043 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:04.455003 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 16 17:56:04.455199 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:04.455033 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Jul 16 17:56:05.137395 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.137366 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:56:05.166976 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.166951 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-proxy-tls\") pod \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\" (UID: \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\") " Jul 16 17:56:05.167105 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.166989 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-kube-rbac-proxy-sar-config\") pod \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\" (UID: \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\") " Jul 16 17:56:05.167105 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.167044 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jm2c8\" (UniqueName: \"kubernetes.io/projected/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-kube-api-access-jm2c8\") pod \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\" (UID: \"b8f32b5c-c76d-40a4-aaf4-a07591e24eed\") " Jul 16 17:56:05.167345 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.167318 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8f32b5c-c76d-40a4-aaf4-a07591e24eed" (UID: "b8f32b5c-c76d-40a4-aaf4-a07591e24eed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:56:05.169114 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.169088 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8f32b5c-c76d-40a4-aaf4-a07591e24eed" (UID: "b8f32b5c-c76d-40a4-aaf4-a07591e24eed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:56:05.169216 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.169190 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-kube-api-access-jm2c8" (OuterVolumeSpecName: "kube-api-access-jm2c8") pod "b8f32b5c-c76d-40a4-aaf4-a07591e24eed" (UID: "b8f32b5c-c76d-40a4-aaf4-a07591e24eed"). InnerVolumeSpecName "kube-api-access-jm2c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:56:05.267779 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.267720 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm2c8\" (UniqueName: \"kubernetes.io/projected/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-kube-api-access-jm2c8\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:56:05.267779 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.267744 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:56:05.267779 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.267753 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8f32b5c-c76d-40a4-aaf4-a07591e24eed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:56:05.459448 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.459414 2582 generic.go:356] "Generic (PLEG): container finished" podID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerID="948cfbe4f60fa65e355e21e32ebdf4971c764d986632747d18bfe67f05f64967" exitCode=0 Jul 16 17:56:05.459745 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.459492 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" Jul 16 17:56:05.459745 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.459512 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" event={"ID":"b8f32b5c-c76d-40a4-aaf4-a07591e24eed","Type":"ContainerDied","Data":"948cfbe4f60fa65e355e21e32ebdf4971c764d986632747d18bfe67f05f64967"} Jul 16 17:56:05.459745 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.459575 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d" event={"ID":"b8f32b5c-c76d-40a4-aaf4-a07591e24eed","Type":"ContainerDied","Data":"ee62a54fbfbecd0cfdaede0da8e16985afc54c2f5e37c03bbd8e10cce6678dc1"} Jul 16 17:56:05.459745 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.459606 2582 scope.go:117] "RemoveContainer" containerID="13395cafa425e20492ac4e2d916037df588821e46d5d6a2f108e31c65d3f5828" Jul 16 17:56:05.471829 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.471809 2582 scope.go:117] "RemoveContainer" containerID="948cfbe4f60fa65e355e21e32ebdf4971c764d986632747d18bfe67f05f64967" Jul 16 17:56:05.479839 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.479821 2582 scope.go:117] "RemoveContainer" containerID="13395cafa425e20492ac4e2d916037df588821e46d5d6a2f108e31c65d3f5828" Jul 16 17:56:05.480159 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:56:05.480132 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13395cafa425e20492ac4e2d916037df588821e46d5d6a2f108e31c65d3f5828\": container with ID starting with 13395cafa425e20492ac4e2d916037df588821e46d5d6a2f108e31c65d3f5828 not found: ID does not exist" containerID="13395cafa425e20492ac4e2d916037df588821e46d5d6a2f108e31c65d3f5828" Jul 16 17:56:05.480253 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.480173 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13395cafa425e20492ac4e2d916037df588821e46d5d6a2f108e31c65d3f5828"} err="failed to get container status \"13395cafa425e20492ac4e2d916037df588821e46d5d6a2f108e31c65d3f5828\": rpc error: code = NotFound desc = could not find container \"13395cafa425e20492ac4e2d916037df588821e46d5d6a2f108e31c65d3f5828\": container with ID starting with 13395cafa425e20492ac4e2d916037df588821e46d5d6a2f108e31c65d3f5828 not found: ID does not exist" Jul 16 17:56:05.480253 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.480200 2582 scope.go:117] "RemoveContainer" containerID="948cfbe4f60fa65e355e21e32ebdf4971c764d986632747d18bfe67f05f64967" Jul 16 17:56:05.480790 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:56:05.480645 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948cfbe4f60fa65e355e21e32ebdf4971c764d986632747d18bfe67f05f64967\": container with ID starting with 948cfbe4f60fa65e355e21e32ebdf4971c764d986632747d18bfe67f05f64967 not found: ID does not exist" containerID="948cfbe4f60fa65e355e21e32ebdf4971c764d986632747d18bfe67f05f64967" Jul 16 17:56:05.480790 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.480677 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948cfbe4f60fa65e355e21e32ebdf4971c764d986632747d18bfe67f05f64967"} err="failed to get container status \"948cfbe4f60fa65e355e21e32ebdf4971c764d986632747d18bfe67f05f64967\": rpc error: code = NotFound desc = could not find container \"948cfbe4f60fa65e355e21e32ebdf4971c764d986632747d18bfe67f05f64967\": container with ID starting with 948cfbe4f60fa65e355e21e32ebdf4971c764d986632747d18bfe67f05f64967 not found: ID does not exist" Jul 16 17:56:05.482036 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.482014 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d"] Jul 16 17:56:05.487161 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.487137 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f0d41-predictor-9cfbdbc84-cjh4d"] Jul 16 17:56:05.594078 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.594056 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:56:05.671368 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.671338 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/773df83f-1eb5-4d26-a88e-d6bd277ad72e-proxy-tls\") pod \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\" (UID: \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\") " Jul 16 17:56:05.671515 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.671386 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbnqw\" (UniqueName: \"kubernetes.io/projected/773df83f-1eb5-4d26-a88e-d6bd277ad72e-kube-api-access-jbnqw\") pod \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\" (UID: \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\") " Jul 16 17:56:05.671515 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.671484 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/773df83f-1eb5-4d26-a88e-d6bd277ad72e-kube-rbac-proxy-sar-config\") pod \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\" (UID: \"773df83f-1eb5-4d26-a88e-d6bd277ad72e\") " Jul 16 17:56:05.671862 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.671834 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/773df83f-1eb5-4d26-a88e-d6bd277ad72e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "773df83f-1eb5-4d26-a88e-d6bd277ad72e" (UID: "773df83f-1eb5-4d26-a88e-d6bd277ad72e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:56:05.673599 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.673572 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773df83f-1eb5-4d26-a88e-d6bd277ad72e-kube-api-access-jbnqw" (OuterVolumeSpecName: "kube-api-access-jbnqw") pod "773df83f-1eb5-4d26-a88e-d6bd277ad72e" (UID: "773df83f-1eb5-4d26-a88e-d6bd277ad72e"). InnerVolumeSpecName "kube-api-access-jbnqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:56:05.673691 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.673603 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/773df83f-1eb5-4d26-a88e-d6bd277ad72e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "773df83f-1eb5-4d26-a88e-d6bd277ad72e" (UID: "773df83f-1eb5-4d26-a88e-d6bd277ad72e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:56:05.772426 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.772402 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/773df83f-1eb5-4d26-a88e-d6bd277ad72e-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:56:05.772426 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.772423 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbnqw\" (UniqueName: \"kubernetes.io/projected/773df83f-1eb5-4d26-a88e-d6bd277ad72e-kube-api-access-jbnqw\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:56:05.772594 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:05.772432 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/773df83f-1eb5-4d26-a88e-d6bd277ad72e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:56:06.465989 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.465902 2582 generic.go:356] "Generic (PLEG): container finished" podID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerID="2244c4063f1e641a5ff34bb725a6fa4b5aa7d3cd73aaf6d8e4a87aaf0ae4ebbf" exitCode=0 Jul 16 17:56:06.465989 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.465954 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" event={"ID":"773df83f-1eb5-4d26-a88e-d6bd277ad72e","Type":"ContainerDied","Data":"2244c4063f1e641a5ff34bb725a6fa4b5aa7d3cd73aaf6d8e4a87aaf0ae4ebbf"} Jul 16 17:56:06.465989 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.465959 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" Jul 16 17:56:06.466490 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.466001 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94" event={"ID":"773df83f-1eb5-4d26-a88e-d6bd277ad72e","Type":"ContainerDied","Data":"a9a09a72d09033259c32b6f5ec8c2672b39ddb1b15df50e241a8817aace87d64"} Jul 16 17:56:06.466490 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.466027 2582 scope.go:117] "RemoveContainer" containerID="898c017e5f1316e86e09b4d03021aecfeb7c582fcb0fb55429853babbc54768c" Jul 16 17:56:06.475114 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.475096 2582 scope.go:117] "RemoveContainer" containerID="2244c4063f1e641a5ff34bb725a6fa4b5aa7d3cd73aaf6d8e4a87aaf0ae4ebbf" Jul 16 17:56:06.482434 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.482407 2582 scope.go:117] "RemoveContainer" containerID="898c017e5f1316e86e09b4d03021aecfeb7c582fcb0fb55429853babbc54768c" Jul 16 17:56:06.482796 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:56:06.482774 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898c017e5f1316e86e09b4d03021aecfeb7c582fcb0fb55429853babbc54768c\": container with ID starting with 898c017e5f1316e86e09b4d03021aecfeb7c582fcb0fb55429853babbc54768c not found: ID does not exist" containerID="898c017e5f1316e86e09b4d03021aecfeb7c582fcb0fb55429853babbc54768c" Jul 16 17:56:06.482896 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.482800 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898c017e5f1316e86e09b4d03021aecfeb7c582fcb0fb55429853babbc54768c"} err="failed to get container status \"898c017e5f1316e86e09b4d03021aecfeb7c582fcb0fb55429853babbc54768c\": rpc error: code = NotFound desc = could not find container \"898c017e5f1316e86e09b4d03021aecfeb7c582fcb0fb55429853babbc54768c\": container with ID starting with 898c017e5f1316e86e09b4d03021aecfeb7c582fcb0fb55429853babbc54768c not found: ID does not exist" Jul 16 17:56:06.482896 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.482817 2582 scope.go:117] "RemoveContainer" containerID="2244c4063f1e641a5ff34bb725a6fa4b5aa7d3cd73aaf6d8e4a87aaf0ae4ebbf" Jul 16 17:56:06.483076 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:56:06.483049 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2244c4063f1e641a5ff34bb725a6fa4b5aa7d3cd73aaf6d8e4a87aaf0ae4ebbf\": container with ID starting with 2244c4063f1e641a5ff34bb725a6fa4b5aa7d3cd73aaf6d8e4a87aaf0ae4ebbf not found: ID does not exist" containerID="2244c4063f1e641a5ff34bb725a6fa4b5aa7d3cd73aaf6d8e4a87aaf0ae4ebbf" Jul 16 17:56:06.483136 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.483086 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2244c4063f1e641a5ff34bb725a6fa4b5aa7d3cd73aaf6d8e4a87aaf0ae4ebbf"} err="failed to get container status \"2244c4063f1e641a5ff34bb725a6fa4b5aa7d3cd73aaf6d8e4a87aaf0ae4ebbf\": rpc error: code = NotFound desc = could not find container \"2244c4063f1e641a5ff34bb725a6fa4b5aa7d3cd73aaf6d8e4a87aaf0ae4ebbf\": container with ID starting with 2244c4063f1e641a5ff34bb725a6fa4b5aa7d3cd73aaf6d8e4a87aaf0ae4ebbf not found: ID does not exist" Jul 16 17:56:06.487626 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.487604 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94"] Jul 16 17:56:06.492077 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.492053 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0d41-predictor-8bd487845-z6k94"] Jul 16 17:56:06.584692 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.584654 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" path="/var/lib/kubelet/pods/773df83f-1eb5-4d26-a88e-d6bd277ad72e/volumes" Jul 16 17:56:06.585073 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:06.585056 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" path="/var/lib/kubelet/pods/b8f32b5c-c76d-40a4-aaf4-a07591e24eed/volumes" Jul 16 17:56:09.459445 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:09.459405 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:09.459862 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:09.459805 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:09.459862 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:09.459847 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Jul 16 17:56:09.460297 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:09.460274 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 16 17:56:19.460451 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:19.460410 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 16 17:56:19.460847 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:19.460410 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Jul 16 17:56:29.460211 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:29.460172 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Jul 16 17:56:29.460611 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:29.460176 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 16 17:56:39.460815 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:39.460736 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 16 17:56:39.461271 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:39.460741 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Jul 16 17:56:41.548994 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.548954 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds"] Jul 16 17:56:41.549438 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.549223 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kserve-container" containerID="cri-o://ba7098c56428e18c52238ddf0982139c8378b0a73a307883d6fb89b0553ea384" gracePeriod=30 Jul 16 17:56:41.549438 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.549267 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kube-rbac-proxy" containerID="cri-o://cdf66a32cfdf7c75b32583ff00a3d22b3235748ddbbbe7a5ad160ae0ff72464d" gracePeriod=30 Jul 16 17:56:41.630715 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.630684 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h"] Jul 16 17:56:41.630962 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.630935 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kserve-container" containerID="cri-o://726202d58c026062b90f00a2ec24cf1b09bf568df87adb09d9e2e61048cf6af6" gracePeriod=30 Jul 16 17:56:41.631060 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.630946 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kube-rbac-proxy" containerID="cri-o://bc8d901a08ba16ed3ee2876249da7c0a14283e0a79e04bf11139779a2dd40fdc" gracePeriod=30 Jul 16 17:56:41.672934 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.672905 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl"] Jul 16 17:56:41.673232 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.673219 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kserve-container" Jul 16 17:56:41.673320 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.673233 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kserve-container" Jul 16 17:56:41.673320 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.673244 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kube-rbac-proxy" Jul 16 17:56:41.673320 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.673249 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kube-rbac-proxy" Jul 16 17:56:41.673320 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.673268 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kube-rbac-proxy" Jul 16 17:56:41.673320 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.673273 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kube-rbac-proxy" Jul 16 17:56:41.673320 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.673280 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kserve-container" Jul 16 17:56:41.673320 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.673285 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kserve-container" Jul 16 17:56:41.673579 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.673335 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kserve-container" Jul 16 17:56:41.673579 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.673342 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8f32b5c-c76d-40a4-aaf4-a07591e24eed" containerName="kube-rbac-proxy" Jul 16 17:56:41.673579 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.673349 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kserve-container" Jul 16 17:56:41.673579 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.673356 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="773df83f-1eb5-4d26-a88e-d6bd277ad72e" containerName="kube-rbac-proxy" Jul 16 17:56:41.676064 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.676046 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:41.678859 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.678637 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6687f-predictor-serving-cert\"" Jul 16 17:56:41.678859 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.678682 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6687f-kube-rbac-proxy-sar-config\"" Jul 16 17:56:41.687701 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.687668 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl"] Jul 16 17:56:41.769204 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.769172 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv"] Jul 16 17:56:41.772328 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.772311 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:41.774750 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.774730 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6687f-predictor-serving-cert\"" Jul 16 17:56:41.774853 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.774793 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6687f-kube-rbac-proxy-sar-config\"" Jul 16 17:56:41.785032 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.785008 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv"] Jul 16 17:56:41.851243 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.851172 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05e1f5e2-49a6-4db2-94df-a43dd7e60837-proxy-tls\") pod \"success-200-isvc-6687f-predictor-769fddcb6f-tfzgl\" (UID: \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\") " pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:41.851243 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.851231 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7k2b\" (UniqueName: \"kubernetes.io/projected/05e1f5e2-49a6-4db2-94df-a43dd7e60837-kube-api-access-d7k2b\") pod \"success-200-isvc-6687f-predictor-769fddcb6f-tfzgl\" (UID: \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\") " pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:41.851390 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.851333 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05e1f5e2-49a6-4db2-94df-a43dd7e60837-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6687f-predictor-769fddcb6f-tfzgl\" (UID: \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\") " pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:41.952650 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.952618 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7k2b\" (UniqueName: \"kubernetes.io/projected/05e1f5e2-49a6-4db2-94df-a43dd7e60837-kube-api-access-d7k2b\") pod \"success-200-isvc-6687f-predictor-769fddcb6f-tfzgl\" (UID: \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\") " pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:41.952780 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.952669 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/822c7304-5d93-443d-aa03-1d20f646a951-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6687f-predictor-547df57596-fmjkv\" (UID: \"822c7304-5d93-443d-aa03-1d20f646a951\") " pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:41.952780 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.952710 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/822c7304-5d93-443d-aa03-1d20f646a951-proxy-tls\") pod \"error-404-isvc-6687f-predictor-547df57596-fmjkv\" (UID: \"822c7304-5d93-443d-aa03-1d20f646a951\") " pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:41.952913 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.952890 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05e1f5e2-49a6-4db2-94df-a43dd7e60837-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6687f-predictor-769fddcb6f-tfzgl\" (UID: \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\") " pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:41.952966 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.952930 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05e1f5e2-49a6-4db2-94df-a43dd7e60837-proxy-tls\") pod \"success-200-isvc-6687f-predictor-769fddcb6f-tfzgl\" (UID: \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\") " pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:41.952966 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.952949 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbx6c\" (UniqueName: \"kubernetes.io/projected/822c7304-5d93-443d-aa03-1d20f646a951-kube-api-access-vbx6c\") pod \"error-404-isvc-6687f-predictor-547df57596-fmjkv\" (UID: \"822c7304-5d93-443d-aa03-1d20f646a951\") " pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:41.953551 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.953530 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05e1f5e2-49a6-4db2-94df-a43dd7e60837-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6687f-predictor-769fddcb6f-tfzgl\" (UID: \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\") " pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:41.955403 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.955386 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05e1f5e2-49a6-4db2-94df-a43dd7e60837-proxy-tls\") pod \"success-200-isvc-6687f-predictor-769fddcb6f-tfzgl\" (UID: \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\") " pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:41.963332 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.963312 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7k2b\" (UniqueName: \"kubernetes.io/projected/05e1f5e2-49a6-4db2-94df-a43dd7e60837-kube-api-access-d7k2b\") pod \"success-200-isvc-6687f-predictor-769fddcb6f-tfzgl\" (UID: \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\") " pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:41.991293 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:41.991263 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:42.054003 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.053968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbx6c\" (UniqueName: \"kubernetes.io/projected/822c7304-5d93-443d-aa03-1d20f646a951-kube-api-access-vbx6c\") pod \"error-404-isvc-6687f-predictor-547df57596-fmjkv\" (UID: \"822c7304-5d93-443d-aa03-1d20f646a951\") " pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:42.054147 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.054063 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/822c7304-5d93-443d-aa03-1d20f646a951-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6687f-predictor-547df57596-fmjkv\" (UID: \"822c7304-5d93-443d-aa03-1d20f646a951\") " pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:42.054147 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.054095 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/822c7304-5d93-443d-aa03-1d20f646a951-proxy-tls\") pod \"error-404-isvc-6687f-predictor-547df57596-fmjkv\" (UID: \"822c7304-5d93-443d-aa03-1d20f646a951\") " pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:42.054927 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.054902 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/822c7304-5d93-443d-aa03-1d20f646a951-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6687f-predictor-547df57596-fmjkv\" (UID: \"822c7304-5d93-443d-aa03-1d20f646a951\") " pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:42.057431 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.057406 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/822c7304-5d93-443d-aa03-1d20f646a951-proxy-tls\") pod \"error-404-isvc-6687f-predictor-547df57596-fmjkv\" (UID: \"822c7304-5d93-443d-aa03-1d20f646a951\") " pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:42.063908 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.063876 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbx6c\" (UniqueName: \"kubernetes.io/projected/822c7304-5d93-443d-aa03-1d20f646a951-kube-api-access-vbx6c\") pod \"error-404-isvc-6687f-predictor-547df57596-fmjkv\" (UID: \"822c7304-5d93-443d-aa03-1d20f646a951\") " pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:42.086100 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.085177 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:42.125455 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.125429 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl"] Jul 16 17:56:42.127448 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:56:42.127420 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e1f5e2_49a6_4db2_94df_a43dd7e60837.slice/crio-a99fade8c5b21f17ad779b1d1f2dc3c32a9720183b015c7e233c1eb6c620c408 WatchSource:0}: Error finding container a99fade8c5b21f17ad779b1d1f2dc3c32a9720183b015c7e233c1eb6c620c408: Status 404 returned error can't find the container with id a99fade8c5b21f17ad779b1d1f2dc3c32a9720183b015c7e233c1eb6c620c408 Jul 16 17:56:42.230269 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.230240 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv"] Jul 16 17:56:42.231546 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:56:42.231525 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822c7304_5d93_443d_aa03_1d20f646a951.slice/crio-b03a2845f03327bcea6f283c60ad68d89816bcd9ac88d87bb0bc24a5e361af38 WatchSource:0}: Error finding container b03a2845f03327bcea6f283c60ad68d89816bcd9ac88d87bb0bc24a5e361af38: Status 404 returned error can't find the container with id b03a2845f03327bcea6f283c60ad68d89816bcd9ac88d87bb0bc24a5e361af38 Jul 16 17:56:42.610179 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.610148 2582 generic.go:356] "Generic (PLEG): container finished" podID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerID="bc8d901a08ba16ed3ee2876249da7c0a14283e0a79e04bf11139779a2dd40fdc" exitCode=2 Jul 16 17:56:42.610569 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.610220 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" event={"ID":"536a3642-b49f-40f3-97f8-9c325a55a02d","Type":"ContainerDied","Data":"bc8d901a08ba16ed3ee2876249da7c0a14283e0a79e04bf11139779a2dd40fdc"} Jul 16 17:56:42.611850 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.611823 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" event={"ID":"822c7304-5d93-443d-aa03-1d20f646a951","Type":"ContainerStarted","Data":"660399f8045b107196b455e25753c0eb1b34582f9398b97e9cb2483a3a499c48"} Jul 16 17:56:42.611989 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.611877 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" event={"ID":"822c7304-5d93-443d-aa03-1d20f646a951","Type":"ContainerStarted","Data":"a4af6de8f18f763ef8dc09186a4dcd71752a3fdc580c79a8c644d6382fb3f987"} Jul 16 17:56:42.611989 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.611908 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" event={"ID":"822c7304-5d93-443d-aa03-1d20f646a951","Type":"ContainerStarted","Data":"b03a2845f03327bcea6f283c60ad68d89816bcd9ac88d87bb0bc24a5e361af38"} Jul 16 17:56:42.611989 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.611942 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:42.611989 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.611980 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:42.613742 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.613715 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" event={"ID":"05e1f5e2-49a6-4db2-94df-a43dd7e60837","Type":"ContainerStarted","Data":"a0e94790de26d695114b9c42e4f07632654aa1e4b93afb759407a1d50e072a1d"} Jul 16 17:56:42.613855 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.613776 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:42.613855 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.613771 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 16 17:56:42.613970 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.613938 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" event={"ID":"05e1f5e2-49a6-4db2-94df-a43dd7e60837","Type":"ContainerStarted","Data":"af79412c85cf4fc455e06c66ed572e27ae1ea521e7811c6ca0a8159935fc2cf6"} Jul 16 17:56:42.614046 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.613990 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" event={"ID":"05e1f5e2-49a6-4db2-94df-a43dd7e60837","Type":"ContainerStarted","Data":"a99fade8c5b21f17ad779b1d1f2dc3c32a9720183b015c7e233c1eb6c620c408"} Jul 16 17:56:42.614046 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.614024 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:42.615070 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.615034 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Jul 16 17:56:42.615654 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.615632 2582 generic.go:356] "Generic (PLEG): container finished" podID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerID="cdf66a32cfdf7c75b32583ff00a3d22b3235748ddbbbe7a5ad160ae0ff72464d" exitCode=2 Jul 16 17:56:42.615742 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.615683 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" event={"ID":"32b3e31a-3cea-433f-84c1-6d4da383701b","Type":"ContainerDied","Data":"cdf66a32cfdf7c75b32583ff00a3d22b3235748ddbbbe7a5ad160ae0ff72464d"} Jul 16 17:56:42.630956 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.630916 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" podStartSLOduration=1.630898057 podStartE2EDuration="1.630898057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:56:42.628932866 +0000 UTC m=+1352.646162207" watchObservedRunningTime="2026-07-16 17:56:42.630898057 +0000 UTC m=+1352.648127399" Jul 16 17:56:42.647842 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:42.647807 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" podStartSLOduration=1.647795929 podStartE2EDuration="1.647795929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:56:42.64608333 +0000 UTC m=+1352.663312671" watchObservedRunningTime="2026-07-16 17:56:42.647795929 +0000 UTC m=+1352.665025324" Jul 16 17:56:43.619970 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:43.619927 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 16 17:56:43.620378 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:43.620049 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Jul 16 17:56:44.625524 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.625492 2582 generic.go:356] "Generic (PLEG): container finished" podID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerID="ba7098c56428e18c52238ddf0982139c8378b0a73a307883d6fb89b0553ea384" exitCode=0 Jul 16 17:56:44.625524 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.625503 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" event={"ID":"32b3e31a-3cea-433f-84c1-6d4da383701b","Type":"ContainerDied","Data":"ba7098c56428e18c52238ddf0982139c8378b0a73a307883d6fb89b0553ea384"} Jul 16 17:56:44.628235 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.628212 2582 generic.go:356] "Generic (PLEG): container finished" podID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerID="726202d58c026062b90f00a2ec24cf1b09bf568df87adb09d9e2e61048cf6af6" exitCode=0 Jul 16 17:56:44.628364 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.628247 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" event={"ID":"536a3642-b49f-40f3-97f8-9c325a55a02d","Type":"ContainerDied","Data":"726202d58c026062b90f00a2ec24cf1b09bf568df87adb09d9e2e61048cf6af6"} Jul 16 17:56:44.714760 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.714740 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:56:44.717500 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.717465 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:56:44.778359 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.778287 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32b3e31a-3cea-433f-84c1-6d4da383701b-kube-rbac-proxy-sar-config\") pod \"32b3e31a-3cea-433f-84c1-6d4da383701b\" (UID: \"32b3e31a-3cea-433f-84c1-6d4da383701b\") " Jul 16 17:56:44.778359 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.778353 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sgqv\" (UniqueName: \"kubernetes.io/projected/32b3e31a-3cea-433f-84c1-6d4da383701b-kube-api-access-2sgqv\") pod \"32b3e31a-3cea-433f-84c1-6d4da383701b\" (UID: \"32b3e31a-3cea-433f-84c1-6d4da383701b\") " Jul 16 17:56:44.778588 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.778394 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb2h9\" (UniqueName: \"kubernetes.io/projected/536a3642-b49f-40f3-97f8-9c325a55a02d-kube-api-access-rb2h9\") pod \"536a3642-b49f-40f3-97f8-9c325a55a02d\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " Jul 16 17:56:44.778708 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.778681 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b3e31a-3cea-433f-84c1-6d4da383701b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32b3e31a-3cea-433f-84c1-6d4da383701b" (UID: "32b3e31a-3cea-433f-84c1-6d4da383701b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:56:44.778867 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.778843 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536a3642-b49f-40f3-97f8-9c325a55a02d-proxy-tls\") pod \"536a3642-b49f-40f3-97f8-9c325a55a02d\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " Jul 16 17:56:44.779730 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.779688 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536a3642-b49f-40f3-97f8-9c325a55a02d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "536a3642-b49f-40f3-97f8-9c325a55a02d" (UID: "536a3642-b49f-40f3-97f8-9c325a55a02d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:56:44.779861 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.779147 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536a3642-b49f-40f3-97f8-9c325a55a02d-kube-rbac-proxy-sar-config\") pod \"536a3642-b49f-40f3-97f8-9c325a55a02d\" (UID: \"536a3642-b49f-40f3-97f8-9c325a55a02d\") " Jul 16 17:56:44.780008 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.779993 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32b3e31a-3cea-433f-84c1-6d4da383701b-proxy-tls\") pod \"32b3e31a-3cea-433f-84c1-6d4da383701b\" (UID: \"32b3e31a-3cea-433f-84c1-6d4da383701b\") " Jul 16 17:56:44.780670 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.780652 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536a3642-b49f-40f3-97f8-9c325a55a02d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:56:44.780806 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.780792 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32b3e31a-3cea-433f-84c1-6d4da383701b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:56:44.781279 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.781250 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536a3642-b49f-40f3-97f8-9c325a55a02d-kube-api-access-rb2h9" (OuterVolumeSpecName: "kube-api-access-rb2h9") pod "536a3642-b49f-40f3-97f8-9c325a55a02d" (UID: "536a3642-b49f-40f3-97f8-9c325a55a02d"). InnerVolumeSpecName "kube-api-access-rb2h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:56:44.781279 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.781240 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b3e31a-3cea-433f-84c1-6d4da383701b-kube-api-access-2sgqv" (OuterVolumeSpecName: "kube-api-access-2sgqv") pod "32b3e31a-3cea-433f-84c1-6d4da383701b" (UID: "32b3e31a-3cea-433f-84c1-6d4da383701b"). InnerVolumeSpecName "kube-api-access-2sgqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:56:44.781722 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.781419 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536a3642-b49f-40f3-97f8-9c325a55a02d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "536a3642-b49f-40f3-97f8-9c325a55a02d" (UID: "536a3642-b49f-40f3-97f8-9c325a55a02d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:56:44.782721 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.782695 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b3e31a-3cea-433f-84c1-6d4da383701b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32b3e31a-3cea-433f-84c1-6d4da383701b" (UID: "32b3e31a-3cea-433f-84c1-6d4da383701b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:56:44.882205 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.882173 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32b3e31a-3cea-433f-84c1-6d4da383701b-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:56:44.882281 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.882209 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sgqv\" (UniqueName: \"kubernetes.io/projected/32b3e31a-3cea-433f-84c1-6d4da383701b-kube-api-access-2sgqv\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:56:44.882281 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.882225 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb2h9\" (UniqueName: \"kubernetes.io/projected/536a3642-b49f-40f3-97f8-9c325a55a02d-kube-api-access-rb2h9\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:56:44.882281 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:44.882239 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536a3642-b49f-40f3-97f8-9c325a55a02d-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:56:45.634120 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:45.634088 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" Jul 16 17:56:45.634556 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:45.634091 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h" event={"ID":"536a3642-b49f-40f3-97f8-9c325a55a02d","Type":"ContainerDied","Data":"a53900cee7a8a61454e94e98055227371006b398d8468f3e66b9254dc19eca4a"} Jul 16 17:56:45.634556 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:45.634238 2582 scope.go:117] "RemoveContainer" containerID="bc8d901a08ba16ed3ee2876249da7c0a14283e0a79e04bf11139779a2dd40fdc" Jul 16 17:56:45.635747 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:45.635726 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" event={"ID":"32b3e31a-3cea-433f-84c1-6d4da383701b","Type":"ContainerDied","Data":"76d338f70a7f0c87c1af7e82e9299eec2fb4f26e145a80c39a27b5b12029ad8e"} Jul 16 17:56:45.635825 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:45.635745 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds" Jul 16 17:56:45.642894 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:45.642811 2582 scope.go:117] "RemoveContainer" containerID="726202d58c026062b90f00a2ec24cf1b09bf568df87adb09d9e2e61048cf6af6" Jul 16 17:56:45.650638 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:45.650622 2582 scope.go:117] "RemoveContainer" containerID="cdf66a32cfdf7c75b32583ff00a3d22b3235748ddbbbe7a5ad160ae0ff72464d" Jul 16 17:56:45.657381 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:45.657363 2582 scope.go:117] "RemoveContainer" containerID="ba7098c56428e18c52238ddf0982139c8378b0a73a307883d6fb89b0553ea384" Jul 16 17:56:45.667259 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:45.667240 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h"] Jul 16 17:56:45.671538 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:45.671520 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-38e88-predictor-5d744d6567-8cj6h"] Jul 16 17:56:45.685023 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:45.684997 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds"] Jul 16 17:56:45.686390 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:45.686370 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-38e88-predictor-859fd4f7c8-bfsds"] Jul 16 17:56:46.586142 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:46.586108 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" path="/var/lib/kubelet/pods/32b3e31a-3cea-433f-84c1-6d4da383701b/volumes" Jul 16 17:56:46.586530 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:46.586512 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" path="/var/lib/kubelet/pods/536a3642-b49f-40f3-97f8-9c325a55a02d/volumes" Jul 16 17:56:48.624722 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:48.624695 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:56:48.625392 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:48.625361 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 16 17:56:48.625845 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:48.625823 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:56:48.626245 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:48.626209 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Jul 16 17:56:49.460671 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:49.460641 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:56:49.460869 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:49.460734 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:56:58.625698 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:58.625644 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 16 17:56:58.626874 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:56:58.626850 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Jul 16 17:57:08.625456 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:08.625422 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 16 17:57:08.626634 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:08.626612 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Jul 16 17:57:12.158425 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.158389 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj"] Jul 16 17:57:12.158865 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.158632 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kserve-container" containerID="cri-o://3042e174bff378095f4cb9a83c70dbdf86b69b9af6504635e64a0fb63a97f724" gracePeriod=30 Jul 16 17:57:12.158865 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.158676 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kube-rbac-proxy" containerID="cri-o://edff043dbc1175b8269ef905b826f5a5721a36728f6332fccd584dfbc30e3bb0" gracePeriod=30 Jul 16 17:57:12.224850 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.224821 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr"] Jul 16 17:57:12.225166 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.225139 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kserve-container" containerID="cri-o://3b0c903fbd29ecb96756ee86f7bc85ed36dc1fa634a376fc2c7cef97fdc8dbb8" gracePeriod=30 Jul 16 17:57:12.225262 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.225189 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kube-rbac-proxy" containerID="cri-o://e663bfb99eff297649c05bfe88696be0ab993ca5ff3335e24904c9b05e8b3288" gracePeriod=30 Jul 16 17:57:12.245648 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.245451 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg"] Jul 16 17:57:12.246070 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.246054 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kserve-container" Jul 16 17:57:12.246070 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.246071 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kserve-container" Jul 16 17:57:12.246190 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.246079 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kube-rbac-proxy" Jul 16 17:57:12.246190 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.246084 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kube-rbac-proxy" Jul 16 17:57:12.246190 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.246092 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kube-rbac-proxy" Jul 16 17:57:12.246190 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.246097 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kube-rbac-proxy" Jul 16 17:57:12.246190 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.246108 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kserve-container" Jul 16 17:57:12.246190 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.246113 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kserve-container" Jul 16 17:57:12.246190 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.246161 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kserve-container" Jul 16 17:57:12.246190 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.246168 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="536a3642-b49f-40f3-97f8-9c325a55a02d" containerName="kube-rbac-proxy" Jul 16 17:57:12.246190 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.246176 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kserve-container" Jul 16 17:57:12.246190 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.246182 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="32b3e31a-3cea-433f-84c1-6d4da383701b" containerName="kube-rbac-proxy" Jul 16 17:57:12.250067 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.250046 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:12.252542 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.252521 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-efe2d-predictor-serving-cert\"" Jul 16 17:57:12.252642 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.252530 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-efe2d-kube-rbac-proxy-sar-config\"" Jul 16 17:57:12.256830 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.256631 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg"] Jul 16 17:57:12.339517 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.339491 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g"] Jul 16 17:57:12.342752 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.342734 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:12.345331 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.345313 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-efe2d-predictor-serving-cert\"" Jul 16 17:57:12.345427 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.345344 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-efe2d-kube-rbac-proxy-sar-config\"" Jul 16 17:57:12.354552 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.354529 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g"] Jul 16 17:57:12.376845 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.376824 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95720232-9c66-4c33-9431-65122d0214fa-proxy-tls\") pod \"success-200-isvc-efe2d-predictor-564c8f4947-m77mg\" (UID: \"95720232-9c66-4c33-9431-65122d0214fa\") " pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:12.376952 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.376857 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-proxy-tls\") pod \"error-404-isvc-efe2d-predictor-568fd6d6fd-br76g\" (UID: \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\") " pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:12.376952 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.376892 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95720232-9c66-4c33-9431-65122d0214fa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-efe2d-predictor-564c8f4947-m77mg\" (UID: \"95720232-9c66-4c33-9431-65122d0214fa\") " pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:12.376952 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.376918 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxwdk\" (UniqueName: \"kubernetes.io/projected/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-kube-api-access-bxwdk\") pod \"error-404-isvc-efe2d-predictor-568fd6d6fd-br76g\" (UID: \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\") " pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:12.377053 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.376969 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb5qq\" (UniqueName: \"kubernetes.io/projected/95720232-9c66-4c33-9431-65122d0214fa-kube-api-access-tb5qq\") pod \"success-200-isvc-efe2d-predictor-564c8f4947-m77mg\" (UID: \"95720232-9c66-4c33-9431-65122d0214fa\") " pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:12.377053 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.377018 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-efe2d-predictor-568fd6d6fd-br76g\" (UID: \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\") " pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:12.477586 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.477502 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxwdk\" (UniqueName: \"kubernetes.io/projected/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-kube-api-access-bxwdk\") pod \"error-404-isvc-efe2d-predictor-568fd6d6fd-br76g\" (UID: \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\") " pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:12.477741 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.477611 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb5qq\" (UniqueName: \"kubernetes.io/projected/95720232-9c66-4c33-9431-65122d0214fa-kube-api-access-tb5qq\") pod \"success-200-isvc-efe2d-predictor-564c8f4947-m77mg\" (UID: \"95720232-9c66-4c33-9431-65122d0214fa\") " pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:12.477741 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.477666 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-efe2d-predictor-568fd6d6fd-br76g\" (UID: \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\") " pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:12.477741 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.477723 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95720232-9c66-4c33-9431-65122d0214fa-proxy-tls\") pod \"success-200-isvc-efe2d-predictor-564c8f4947-m77mg\" (UID: \"95720232-9c66-4c33-9431-65122d0214fa\") " pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:12.477909 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.477747 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-proxy-tls\") pod \"error-404-isvc-efe2d-predictor-568fd6d6fd-br76g\" (UID: \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\") " pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:12.477909 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.477776 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95720232-9c66-4c33-9431-65122d0214fa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-efe2d-predictor-564c8f4947-m77mg\" (UID: \"95720232-9c66-4c33-9431-65122d0214fa\") " pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:12.478747 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.478717 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95720232-9c66-4c33-9431-65122d0214fa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-efe2d-predictor-564c8f4947-m77mg\" (UID: \"95720232-9c66-4c33-9431-65122d0214fa\") " pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:12.479347 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.479312 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-efe2d-predictor-568fd6d6fd-br76g\" (UID: \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\") " pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:12.480844 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.480817 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-proxy-tls\") pod \"error-404-isvc-efe2d-predictor-568fd6d6fd-br76g\" (UID: \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\") " pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:12.481280 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.481256 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95720232-9c66-4c33-9431-65122d0214fa-proxy-tls\") pod \"success-200-isvc-efe2d-predictor-564c8f4947-m77mg\" (UID: \"95720232-9c66-4c33-9431-65122d0214fa\") " pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:12.486698 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.486673 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb5qq\" (UniqueName: \"kubernetes.io/projected/95720232-9c66-4c33-9431-65122d0214fa-kube-api-access-tb5qq\") pod \"success-200-isvc-efe2d-predictor-564c8f4947-m77mg\" (UID: \"95720232-9c66-4c33-9431-65122d0214fa\") " pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:12.487025 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.487003 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxwdk\" (UniqueName: \"kubernetes.io/projected/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-kube-api-access-bxwdk\") pod \"error-404-isvc-efe2d-predictor-568fd6d6fd-br76g\" (UID: \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\") " pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:12.560484 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.560451 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:12.652927 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.652900 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:12.693316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.693187 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg"] Jul 16 17:57:12.696670 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:57:12.696645 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95720232_9c66_4c33_9431_65122d0214fa.slice/crio-aa0f23c8fea959deeea6e8d8a14dc0c719cfb19ccfbd52747323e64979ef40d9 WatchSource:0}: Error finding container aa0f23c8fea959deeea6e8d8a14dc0c719cfb19ccfbd52747323e64979ef40d9: Status 404 returned error can't find the container with id aa0f23c8fea959deeea6e8d8a14dc0c719cfb19ccfbd52747323e64979ef40d9 Jul 16 17:57:12.740642 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.740610 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" event={"ID":"95720232-9c66-4c33-9431-65122d0214fa","Type":"ContainerStarted","Data":"aa0f23c8fea959deeea6e8d8a14dc0c719cfb19ccfbd52747323e64979ef40d9"} Jul 16 17:57:12.742310 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.742288 2582 generic.go:356] "Generic (PLEG): container finished" podID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerID="e663bfb99eff297649c05bfe88696be0ab993ca5ff3335e24904c9b05e8b3288" exitCode=2 Jul 16 17:57:12.742406 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.742352 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" event={"ID":"b1e18acf-38af-4f3e-87db-3375d62803a9","Type":"ContainerDied","Data":"e663bfb99eff297649c05bfe88696be0ab993ca5ff3335e24904c9b05e8b3288"} Jul 16 17:57:12.744284 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.744256 2582 generic.go:356] "Generic (PLEG): container finished" podID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerID="edff043dbc1175b8269ef905b826f5a5721a36728f6332fccd584dfbc30e3bb0" exitCode=2 Jul 16 17:57:12.744364 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.744298 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" event={"ID":"d9452ec7-5072-4e69-ab92-fd9c40131e0a","Type":"ContainerDied","Data":"edff043dbc1175b8269ef905b826f5a5721a36728f6332fccd584dfbc30e3bb0"} Jul 16 17:57:12.793938 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:12.793909 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g"] Jul 16 17:57:12.806493 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:57:12.806435 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704b81b9_c1f6_45c8_9b9e_406696fcb9b5.slice/crio-51fc0aa025e0b201a9c75e92a297f88d2f84faa8a4e92d19f97c505ea7142f65 WatchSource:0}: Error finding container 51fc0aa025e0b201a9c75e92a297f88d2f84faa8a4e92d19f97c505ea7142f65: Status 404 returned error can't find the container with id 51fc0aa025e0b201a9c75e92a297f88d2f84faa8a4e92d19f97c505ea7142f65 Jul 16 17:57:13.750208 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:13.750173 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" event={"ID":"95720232-9c66-4c33-9431-65122d0214fa","Type":"ContainerStarted","Data":"f023cb81c5eaf70d3561f09fdd9eac41e6ba7a800782e9eb34cc7d4b44ab330e"} Jul 16 17:57:13.750669 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:13.750242 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" event={"ID":"95720232-9c66-4c33-9431-65122d0214fa","Type":"ContainerStarted","Data":"1fc5d421834cb00771549beb7dc5232c36b71546f5f4d678851215e0f82555ce"} Jul 16 17:57:13.750669 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:13.750279 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:13.750669 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:13.750308 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:13.751941 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:13.751916 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 16 17:57:13.752068 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:13.752010 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" event={"ID":"704b81b9-c1f6-45c8-9b9e-406696fcb9b5","Type":"ContainerStarted","Data":"1c2907ba69ad6d07581c7f3df5aa81885ed2d8e3c216d464b8e246d19b2d9335"} Jul 16 17:57:13.752132 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:13.752067 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" event={"ID":"704b81b9-c1f6-45c8-9b9e-406696fcb9b5","Type":"ContainerStarted","Data":"d0e4bc5e29093d6679ce8b4697a91a16e4f0e12d4442935094220842d6c007d2"} Jul 16 17:57:13.752132 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:13.752100 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" event={"ID":"704b81b9-c1f6-45c8-9b9e-406696fcb9b5","Type":"ContainerStarted","Data":"51fc0aa025e0b201a9c75e92a297f88d2f84faa8a4e92d19f97c505ea7142f65"} Jul 16 17:57:13.752214 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:13.752193 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:13.769427 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:13.769376 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" podStartSLOduration=1.769360671 podStartE2EDuration="1.769360671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:57:13.768270879 +0000 UTC m=+1383.785500221" watchObservedRunningTime="2026-07-16 17:57:13.769360671 +0000 UTC m=+1383.786590017" Jul 16 17:57:13.789363 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:13.789318 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" podStartSLOduration=1.789303691 podStartE2EDuration="1.789303691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:57:13.788543949 +0000 UTC m=+1383.805773312" watchObservedRunningTime="2026-07-16 17:57:13.789303691 +0000 UTC m=+1383.806533032" Jul 16 17:57:14.455783 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:14.455748 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused" Jul 16 17:57:14.455962 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:14.455749 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused" Jul 16 17:57:14.756283 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:14.756254 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:14.756668 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:14.756297 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 16 17:57:14.757730 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:14.757702 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Jul 16 17:57:15.768048 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:15.768012 2582 generic.go:356] "Generic (PLEG): container finished" podID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerID="3042e174bff378095f4cb9a83c70dbdf86b69b9af6504635e64a0fb63a97f724" exitCode=0 Jul 16 17:57:15.768413 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:15.768080 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" event={"ID":"d9452ec7-5072-4e69-ab92-fd9c40131e0a","Type":"ContainerDied","Data":"3042e174bff378095f4cb9a83c70dbdf86b69b9af6504635e64a0fb63a97f724"} Jul 16 17:57:15.768413 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:15.768370 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Jul 16 17:57:15.860220 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:15.860199 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:57:15.907668 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:15.907643 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9452ec7-5072-4e69-ab92-fd9c40131e0a-kube-rbac-proxy-sar-config\") pod \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\" (UID: \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\") " Jul 16 17:57:15.907786 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:15.907735 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9452ec7-5072-4e69-ab92-fd9c40131e0a-proxy-tls\") pod \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\" (UID: \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\") " Jul 16 17:57:15.907786 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:15.907769 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgzj5\" (UniqueName: \"kubernetes.io/projected/d9452ec7-5072-4e69-ab92-fd9c40131e0a-kube-api-access-kgzj5\") pod \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\" (UID: \"d9452ec7-5072-4e69-ab92-fd9c40131e0a\") " Jul 16 17:57:15.907923 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:15.907904 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9452ec7-5072-4e69-ab92-fd9c40131e0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9452ec7-5072-4e69-ab92-fd9c40131e0a" (UID: "d9452ec7-5072-4e69-ab92-fd9c40131e0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:57:15.909750 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:15.909730 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9452ec7-5072-4e69-ab92-fd9c40131e0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9452ec7-5072-4e69-ab92-fd9c40131e0a" (UID: "d9452ec7-5072-4e69-ab92-fd9c40131e0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:57:15.909880 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:15.909861 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9452ec7-5072-4e69-ab92-fd9c40131e0a-kube-api-access-kgzj5" (OuterVolumeSpecName: "kube-api-access-kgzj5") pod "d9452ec7-5072-4e69-ab92-fd9c40131e0a" (UID: "d9452ec7-5072-4e69-ab92-fd9c40131e0a"). InnerVolumeSpecName "kube-api-access-kgzj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:57:16.009121 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.009094 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9452ec7-5072-4e69-ab92-fd9c40131e0a-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:57:16.009121 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.009118 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgzj5\" (UniqueName: \"kubernetes.io/projected/d9452ec7-5072-4e69-ab92-fd9c40131e0a-kube-api-access-kgzj5\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:57:16.009256 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.009128 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9452ec7-5072-4e69-ab92-fd9c40131e0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:57:16.294138 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.294113 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:57:16.412672 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.412629 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e18acf-38af-4f3e-87db-3375d62803a9-proxy-tls\") pod \"b1e18acf-38af-4f3e-87db-3375d62803a9\" (UID: \"b1e18acf-38af-4f3e-87db-3375d62803a9\") " Jul 16 17:57:16.412814 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.412722 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgqx8\" (UniqueName: \"kubernetes.io/projected/b1e18acf-38af-4f3e-87db-3375d62803a9-kube-api-access-pgqx8\") pod \"b1e18acf-38af-4f3e-87db-3375d62803a9\" (UID: \"b1e18acf-38af-4f3e-87db-3375d62803a9\") " Jul 16 17:57:16.412814 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.412760 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1e18acf-38af-4f3e-87db-3375d62803a9-kube-rbac-proxy-sar-config\") pod \"b1e18acf-38af-4f3e-87db-3375d62803a9\" (UID: \"b1e18acf-38af-4f3e-87db-3375d62803a9\") " Jul 16 17:57:16.413178 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.413149 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e18acf-38af-4f3e-87db-3375d62803a9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1e18acf-38af-4f3e-87db-3375d62803a9" (UID: "b1e18acf-38af-4f3e-87db-3375d62803a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:57:16.414822 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.414794 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e18acf-38af-4f3e-87db-3375d62803a9-kube-api-access-pgqx8" (OuterVolumeSpecName: "kube-api-access-pgqx8") pod "b1e18acf-38af-4f3e-87db-3375d62803a9" (UID: "b1e18acf-38af-4f3e-87db-3375d62803a9"). InnerVolumeSpecName "kube-api-access-pgqx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:57:16.414822 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.414803 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e18acf-38af-4f3e-87db-3375d62803a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1e18acf-38af-4f3e-87db-3375d62803a9" (UID: "b1e18acf-38af-4f3e-87db-3375d62803a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:57:16.513542 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.513517 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1e18acf-38af-4f3e-87db-3375d62803a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:57:16.513542 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.513539 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e18acf-38af-4f3e-87db-3375d62803a9-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:57:16.513750 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.513550 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgqx8\" (UniqueName: \"kubernetes.io/projected/b1e18acf-38af-4f3e-87db-3375d62803a9-kube-api-access-pgqx8\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:57:16.772998 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.772919 2582 generic.go:356] "Generic (PLEG): container finished" podID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerID="3b0c903fbd29ecb96756ee86f7bc85ed36dc1fa634a376fc2c7cef97fdc8dbb8" exitCode=0 Jul 16 17:57:16.772998 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.772968 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" Jul 16 17:57:16.772998 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.772984 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" event={"ID":"b1e18acf-38af-4f3e-87db-3375d62803a9","Type":"ContainerDied","Data":"3b0c903fbd29ecb96756ee86f7bc85ed36dc1fa634a376fc2c7cef97fdc8dbb8"} Jul 16 17:57:16.773499 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.773042 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr" event={"ID":"b1e18acf-38af-4f3e-87db-3375d62803a9","Type":"ContainerDied","Data":"00168229cd58cbc47328ecbdae5fc94574c8c2ab6d44bacbf7d66b9c03a7be92"} Jul 16 17:57:16.773499 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.773078 2582 scope.go:117] "RemoveContainer" containerID="e663bfb99eff297649c05bfe88696be0ab993ca5ff3335e24904c9b05e8b3288" Jul 16 17:57:16.774818 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.774783 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" event={"ID":"d9452ec7-5072-4e69-ab92-fd9c40131e0a","Type":"ContainerDied","Data":"0db49cc7ae2e40bbfdf0e66f97a17d7ba574554c241022612e30bddde3bd3eab"} Jul 16 17:57:16.774818 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.774801 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj" Jul 16 17:57:16.782252 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.782232 2582 scope.go:117] "RemoveContainer" containerID="3b0c903fbd29ecb96756ee86f7bc85ed36dc1fa634a376fc2c7cef97fdc8dbb8" Jul 16 17:57:16.791909 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.791891 2582 scope.go:117] "RemoveContainer" containerID="e663bfb99eff297649c05bfe88696be0ab993ca5ff3335e24904c9b05e8b3288" Jul 16 17:57:16.792404 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:57:16.792384 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e663bfb99eff297649c05bfe88696be0ab993ca5ff3335e24904c9b05e8b3288\": container with ID starting with e663bfb99eff297649c05bfe88696be0ab993ca5ff3335e24904c9b05e8b3288 not found: ID does not exist" containerID="e663bfb99eff297649c05bfe88696be0ab993ca5ff3335e24904c9b05e8b3288" Jul 16 17:57:16.792504 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.792413 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e663bfb99eff297649c05bfe88696be0ab993ca5ff3335e24904c9b05e8b3288"} err="failed to get container status \"e663bfb99eff297649c05bfe88696be0ab993ca5ff3335e24904c9b05e8b3288\": rpc error: code = NotFound desc = could not find container \"e663bfb99eff297649c05bfe88696be0ab993ca5ff3335e24904c9b05e8b3288\": container with ID starting with e663bfb99eff297649c05bfe88696be0ab993ca5ff3335e24904c9b05e8b3288 not found: ID does not exist" Jul 16 17:57:16.792504 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.792431 2582 scope.go:117] "RemoveContainer" containerID="3b0c903fbd29ecb96756ee86f7bc85ed36dc1fa634a376fc2c7cef97fdc8dbb8" Jul 16 17:57:16.792779 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:57:16.792747 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0c903fbd29ecb96756ee86f7bc85ed36dc1fa634a376fc2c7cef97fdc8dbb8\": container with ID starting with 3b0c903fbd29ecb96756ee86f7bc85ed36dc1fa634a376fc2c7cef97fdc8dbb8 not found: ID does not exist" containerID="3b0c903fbd29ecb96756ee86f7bc85ed36dc1fa634a376fc2c7cef97fdc8dbb8" Jul 16 17:57:16.792818 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.792789 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0c903fbd29ecb96756ee86f7bc85ed36dc1fa634a376fc2c7cef97fdc8dbb8"} err="failed to get container status \"3b0c903fbd29ecb96756ee86f7bc85ed36dc1fa634a376fc2c7cef97fdc8dbb8\": rpc error: code = NotFound desc = could not find container \"3b0c903fbd29ecb96756ee86f7bc85ed36dc1fa634a376fc2c7cef97fdc8dbb8\": container with ID starting with 3b0c903fbd29ecb96756ee86f7bc85ed36dc1fa634a376fc2c7cef97fdc8dbb8 not found: ID does not exist" Jul 16 17:57:16.792818 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.792811 2582 scope.go:117] "RemoveContainer" containerID="edff043dbc1175b8269ef905b826f5a5721a36728f6332fccd584dfbc30e3bb0" Jul 16 17:57:16.797170 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.797147 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr"] Jul 16 17:57:16.800847 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.800831 2582 scope.go:117] "RemoveContainer" containerID="3042e174bff378095f4cb9a83c70dbdf86b69b9af6504635e64a0fb63a97f724" Jul 16 17:57:16.801373 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.801356 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1da67-predictor-5c69b7464d-wgnzr"] Jul 16 17:57:16.812507 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.812488 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj"] Jul 16 17:57:16.817049 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:16.817032 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1da67-predictor-76bff85dd9-db9wj"] Jul 16 17:57:18.587687 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:18.587655 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" path="/var/lib/kubelet/pods/b1e18acf-38af-4f3e-87db-3375d62803a9/volumes" Jul 16 17:57:18.588092 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:18.588078 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" path="/var/lib/kubelet/pods/d9452ec7-5072-4e69-ab92-fd9c40131e0a/volumes" Jul 16 17:57:18.625524 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:18.625496 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 16 17:57:18.626652 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:18.626629 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused" Jul 16 17:57:19.761329 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:19.761296 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:19.761938 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:19.761909 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 16 17:57:20.773451 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:20.773422 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:57:20.773931 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:20.773907 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Jul 16 17:57:28.626448 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:28.626420 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:57:28.627170 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:28.627151 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:57:29.762872 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:29.762833 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 16 17:57:30.774864 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:30.774818 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Jul 16 17:57:39.762633 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:39.762595 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 16 17:57:40.773879 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:40.773839 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Jul 16 17:57:49.762593 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:49.762553 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 16 17:57:50.774492 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:50.774441 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Jul 16 17:57:51.849308 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.849264 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv"] Jul 16 17:57:51.849759 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.849528 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kserve-container" containerID="cri-o://a4af6de8f18f763ef8dc09186a4dcd71752a3fdc580c79a8c644d6382fb3f987" gracePeriod=30 Jul 16 17:57:51.849759 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.849593 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kube-rbac-proxy" containerID="cri-o://660399f8045b107196b455e25753c0eb1b34582f9398b97e9cb2483a3a499c48" gracePeriod=30 Jul 16 17:57:51.900727 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.900699 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl"] Jul 16 17:57:51.901047 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.900916 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kserve-container" containerID="cri-o://af79412c85cf4fc455e06c66ed572e27ae1ea521e7811c6ca0a8159935fc2cf6" gracePeriod=30 Jul 16 17:57:51.901047 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.900961 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kube-rbac-proxy" containerID="cri-o://a0e94790de26d695114b9c42e4f07632654aa1e4b93afb759407a1d50e072a1d" gracePeriod=30 Jul 16 17:57:51.937482 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.937451 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft"] Jul 16 17:57:51.937792 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.937779 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kube-rbac-proxy" Jul 16 17:57:51.937836 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.937794 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kube-rbac-proxy" Jul 16 17:57:51.937836 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.937805 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kube-rbac-proxy" Jul 16 17:57:51.937836 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.937812 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kube-rbac-proxy" Jul 16 17:57:51.937836 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.937823 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kserve-container" Jul 16 17:57:51.937836 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.937828 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kserve-container" Jul 16 17:57:51.937981 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.937841 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kserve-container" Jul 16 17:57:51.937981 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.937846 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kserve-container" Jul 16 17:57:51.937981 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.937892 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kserve-container" Jul 16 17:57:51.937981 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.937900 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kserve-container" Jul 16 17:57:51.937981 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.937907 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9452ec7-5072-4e69-ab92-fd9c40131e0a" containerName="kube-rbac-proxy" Jul 16 17:57:51.937981 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.937913 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e18acf-38af-4f3e-87db-3375d62803a9" containerName="kube-rbac-proxy" Jul 16 17:57:51.941217 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.941200 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:51.943662 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.943639 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a90bb-predictor-serving-cert\"" Jul 16 17:57:51.943758 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.943643 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a90bb-kube-rbac-proxy-sar-config\"" Jul 16 17:57:51.949791 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.949769 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft"] Jul 16 17:57:51.985198 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.985166 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d38b5b-b79d-496b-bbd5-f0baa7173598-proxy-tls\") pod \"success-200-isvc-a90bb-predictor-675fc99bf9-zwxft\" (UID: \"77d38b5b-b79d-496b-bbd5-f0baa7173598\") " pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:51.985355 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.985257 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxxll\" (UniqueName: \"kubernetes.io/projected/77d38b5b-b79d-496b-bbd5-f0baa7173598-kube-api-access-qxxll\") pod \"success-200-isvc-a90bb-predictor-675fc99bf9-zwxft\" (UID: \"77d38b5b-b79d-496b-bbd5-f0baa7173598\") " pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:51.985355 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:51.985299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d38b5b-b79d-496b-bbd5-f0baa7173598-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a90bb-predictor-675fc99bf9-zwxft\" (UID: \"77d38b5b-b79d-496b-bbd5-f0baa7173598\") " pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:52.054371 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.054344 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb"] Jul 16 17:57:52.057696 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.057678 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:52.060291 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.060258 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a90bb-kube-rbac-proxy-sar-config\"" Jul 16 17:57:52.060291 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.060277 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a90bb-predictor-serving-cert\"" Jul 16 17:57:52.066772 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.066751 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb"] Jul 16 17:57:52.086187 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.086147 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxxll\" (UniqueName: \"kubernetes.io/projected/77d38b5b-b79d-496b-bbd5-f0baa7173598-kube-api-access-qxxll\") pod \"success-200-isvc-a90bb-predictor-675fc99bf9-zwxft\" (UID: \"77d38b5b-b79d-496b-bbd5-f0baa7173598\") " pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:52.086318 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.086198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d38b5b-b79d-496b-bbd5-f0baa7173598-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a90bb-predictor-675fc99bf9-zwxft\" (UID: \"77d38b5b-b79d-496b-bbd5-f0baa7173598\") " pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:52.086318 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.086245 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d38b5b-b79d-496b-bbd5-f0baa7173598-proxy-tls\") pod \"success-200-isvc-a90bb-predictor-675fc99bf9-zwxft\" (UID: \"77d38b5b-b79d-496b-bbd5-f0baa7173598\") " pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:52.086318 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.086277 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lspcn\" (UniqueName: \"kubernetes.io/projected/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-kube-api-access-lspcn\") pod \"error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb\" (UID: \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\") " pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:52.086318 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.086312 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-proxy-tls\") pod \"error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb\" (UID: \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\") " pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:52.086566 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.086339 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb\" (UID: \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\") " pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:52.087082 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.087058 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d38b5b-b79d-496b-bbd5-f0baa7173598-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a90bb-predictor-675fc99bf9-zwxft\" (UID: \"77d38b5b-b79d-496b-bbd5-f0baa7173598\") " pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:52.089485 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.089453 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d38b5b-b79d-496b-bbd5-f0baa7173598-proxy-tls\") pod \"success-200-isvc-a90bb-predictor-675fc99bf9-zwxft\" (UID: \"77d38b5b-b79d-496b-bbd5-f0baa7173598\") " pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:52.094962 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.094934 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxxll\" (UniqueName: \"kubernetes.io/projected/77d38b5b-b79d-496b-bbd5-f0baa7173598-kube-api-access-qxxll\") pod \"success-200-isvc-a90bb-predictor-675fc99bf9-zwxft\" (UID: \"77d38b5b-b79d-496b-bbd5-f0baa7173598\") " pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:52.186845 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.186767 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lspcn\" (UniqueName: \"kubernetes.io/projected/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-kube-api-access-lspcn\") pod \"error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb\" (UID: \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\") " pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:52.186845 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.186825 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-proxy-tls\") pod \"error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb\" (UID: \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\") " pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:52.187016 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.186855 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb\" (UID: \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\") " pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:52.187748 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.187726 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb\" (UID: \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\") " pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:52.189270 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.189253 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-proxy-tls\") pod \"error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb\" (UID: \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\") " pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:52.200258 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.200232 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lspcn\" (UniqueName: \"kubernetes.io/projected/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-kube-api-access-lspcn\") pod \"error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb\" (UID: \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\") " pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:52.251267 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.251245 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:52.368021 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.367988 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:52.380413 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.380388 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft"] Jul 16 17:57:52.392058 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:57:52.392026 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d38b5b_b79d_496b_bbd5_f0baa7173598.slice/crio-6bf966943b2e81f0cc2ced25b1476f925233b5a71f8f8d29926a98c19bb03491 WatchSource:0}: Error finding container 6bf966943b2e81f0cc2ced25b1476f925233b5a71f8f8d29926a98c19bb03491: Status 404 returned error can't find the container with id 6bf966943b2e81f0cc2ced25b1476f925233b5a71f8f8d29926a98c19bb03491 Jul 16 17:57:52.510925 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.510904 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb"] Jul 16 17:57:52.512914 ip-10-0-139-103 kubenswrapper[2582]: W0716 17:57:52.512891 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad71ae1_a1c9_40ad_9fc8_5f0733be67bd.slice/crio-858ae080a50577373c2509c4a351a2a6e3160b299009a74920774e22b3b33c7d WatchSource:0}: Error finding container 858ae080a50577373c2509c4a351a2a6e3160b299009a74920774e22b3b33c7d: Status 404 returned error can't find the container with id 858ae080a50577373c2509c4a351a2a6e3160b299009a74920774e22b3b33c7d Jul 16 17:57:52.923554 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.923515 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" event={"ID":"77d38b5b-b79d-496b-bbd5-f0baa7173598","Type":"ContainerStarted","Data":"62d907ad1985c73f31ca0ec56e82d1d41356fb49956f4c637ce4acd9c332daaa"} Jul 16 17:57:52.923936 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.923580 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" event={"ID":"77d38b5b-b79d-496b-bbd5-f0baa7173598","Type":"ContainerStarted","Data":"b2fc8acb10c85af8bcaf7ed1244ad399a7f3d8cd5f9c55b827876f711eb8067c"} Jul 16 17:57:52.923936 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.923604 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" event={"ID":"77d38b5b-b79d-496b-bbd5-f0baa7173598","Type":"ContainerStarted","Data":"6bf966943b2e81f0cc2ced25b1476f925233b5a71f8f8d29926a98c19bb03491"} Jul 16 17:57:52.923936 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.923636 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:52.925297 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.925276 2582 generic.go:356] "Generic (PLEG): container finished" podID="822c7304-5d93-443d-aa03-1d20f646a951" containerID="660399f8045b107196b455e25753c0eb1b34582f9398b97e9cb2483a3a499c48" exitCode=2 Jul 16 17:57:52.925372 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.925351 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" event={"ID":"822c7304-5d93-443d-aa03-1d20f646a951","Type":"ContainerDied","Data":"660399f8045b107196b455e25753c0eb1b34582f9398b97e9cb2483a3a499c48"} Jul 16 17:57:52.926914 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.926893 2582 generic.go:356] "Generic (PLEG): container finished" podID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerID="a0e94790de26d695114b9c42e4f07632654aa1e4b93afb759407a1d50e072a1d" exitCode=2 Jul 16 17:57:52.927035 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.926956 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" event={"ID":"05e1f5e2-49a6-4db2-94df-a43dd7e60837","Type":"ContainerDied","Data":"a0e94790de26d695114b9c42e4f07632654aa1e4b93afb759407a1d50e072a1d"} Jul 16 17:57:52.928659 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.928633 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" event={"ID":"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd","Type":"ContainerStarted","Data":"b6d9ed67ed4bf44bb46418bd953b5a327115b45cae610fd73bfca408cf6c40ce"} Jul 16 17:57:52.928764 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.928689 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" event={"ID":"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd","Type":"ContainerStarted","Data":"6af61e574e20b7428a0bd93f1c06b9b8265144c16e26dc31c1db2a498beaf116"} Jul 16 17:57:52.928764 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.928726 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:52.928764 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.928755 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" event={"ID":"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd","Type":"ContainerStarted","Data":"858ae080a50577373c2509c4a351a2a6e3160b299009a74920774e22b3b33c7d"} Jul 16 17:57:52.928895 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.928789 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:52.930101 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.930079 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 16 17:57:52.941767 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.941733 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" podStartSLOduration=1.941721399 podStartE2EDuration="1.941721399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:57:52.940194193 +0000 UTC m=+1422.957423534" watchObservedRunningTime="2026-07-16 17:57:52.941721399 +0000 UTC m=+1422.958950740" Jul 16 17:57:52.957824 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:52.957790 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" podStartSLOduration=0.957775117 podStartE2EDuration="957.775117ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:57:52.95710377 +0000 UTC m=+1422.974333113" watchObservedRunningTime="2026-07-16 17:57:52.957775117 +0000 UTC m=+1422.975004459" Jul 16 17:57:53.620010 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:53.619963 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Jul 16 17:57:53.620176 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:53.620002 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused" Jul 16 17:57:53.933424 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:53.933330 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 16 17:57:53.933424 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:53.933409 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:53.934604 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:53.934581 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 16 17:57:54.940569 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:54.940536 2582 generic.go:356] "Generic (PLEG): container finished" podID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerID="af79412c85cf4fc455e06c66ed572e27ae1ea521e7811c6ca0a8159935fc2cf6" exitCode=0 Jul 16 17:57:54.941035 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:54.940606 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" event={"ID":"05e1f5e2-49a6-4db2-94df-a43dd7e60837","Type":"ContainerDied","Data":"af79412c85cf4fc455e06c66ed572e27ae1ea521e7811c6ca0a8159935fc2cf6"} Jul 16 17:57:54.941035 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:54.940871 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 16 17:57:55.042438 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.042409 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:57:55.110382 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.110354 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05e1f5e2-49a6-4db2-94df-a43dd7e60837-proxy-tls\") pod \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\" (UID: \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\") " Jul 16 17:57:55.110539 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.110398 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7k2b\" (UniqueName: \"kubernetes.io/projected/05e1f5e2-49a6-4db2-94df-a43dd7e60837-kube-api-access-d7k2b\") pod \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\" (UID: \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\") " Jul 16 17:57:55.110539 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.110423 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05e1f5e2-49a6-4db2-94df-a43dd7e60837-kube-rbac-proxy-sar-config\") pod \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\" (UID: \"05e1f5e2-49a6-4db2-94df-a43dd7e60837\") " Jul 16 17:57:55.110815 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.110793 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e1f5e2-49a6-4db2-94df-a43dd7e60837-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "05e1f5e2-49a6-4db2-94df-a43dd7e60837" (UID: "05e1f5e2-49a6-4db2-94df-a43dd7e60837"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:57:55.112609 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.112583 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e1f5e2-49a6-4db2-94df-a43dd7e60837-kube-api-access-d7k2b" (OuterVolumeSpecName: "kube-api-access-d7k2b") pod "05e1f5e2-49a6-4db2-94df-a43dd7e60837" (UID: "05e1f5e2-49a6-4db2-94df-a43dd7e60837"). InnerVolumeSpecName "kube-api-access-d7k2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:57:55.112688 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.112665 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05e1f5e2-49a6-4db2-94df-a43dd7e60837-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "05e1f5e2-49a6-4db2-94df-a43dd7e60837" (UID: "05e1f5e2-49a6-4db2-94df-a43dd7e60837"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:57:55.211301 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.211218 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05e1f5e2-49a6-4db2-94df-a43dd7e60837-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:57:55.211301 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.211256 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7k2b\" (UniqueName: \"kubernetes.io/projected/05e1f5e2-49a6-4db2-94df-a43dd7e60837-kube-api-access-d7k2b\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:57:55.211301 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.211270 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05e1f5e2-49a6-4db2-94df-a43dd7e60837-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:57:55.789042 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.789013 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:57:55.816548 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.816510 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/822c7304-5d93-443d-aa03-1d20f646a951-kube-rbac-proxy-sar-config\") pod \"822c7304-5d93-443d-aa03-1d20f646a951\" (UID: \"822c7304-5d93-443d-aa03-1d20f646a951\") " Jul 16 17:57:55.816694 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.816569 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/822c7304-5d93-443d-aa03-1d20f646a951-proxy-tls\") pod \"822c7304-5d93-443d-aa03-1d20f646a951\" (UID: \"822c7304-5d93-443d-aa03-1d20f646a951\") " Jul 16 17:57:55.816694 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.816595 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbx6c\" (UniqueName: \"kubernetes.io/projected/822c7304-5d93-443d-aa03-1d20f646a951-kube-api-access-vbx6c\") pod \"822c7304-5d93-443d-aa03-1d20f646a951\" (UID: \"822c7304-5d93-443d-aa03-1d20f646a951\") " Jul 16 17:57:55.816954 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.816921 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822c7304-5d93-443d-aa03-1d20f646a951-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "822c7304-5d93-443d-aa03-1d20f646a951" (UID: "822c7304-5d93-443d-aa03-1d20f646a951"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:57:55.818649 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.818621 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822c7304-5d93-443d-aa03-1d20f646a951-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "822c7304-5d93-443d-aa03-1d20f646a951" (UID: "822c7304-5d93-443d-aa03-1d20f646a951"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:57:55.818733 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.818710 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822c7304-5d93-443d-aa03-1d20f646a951-kube-api-access-vbx6c" (OuterVolumeSpecName: "kube-api-access-vbx6c") pod "822c7304-5d93-443d-aa03-1d20f646a951" (UID: "822c7304-5d93-443d-aa03-1d20f646a951"). InnerVolumeSpecName "kube-api-access-vbx6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:57:55.917977 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.917940 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/822c7304-5d93-443d-aa03-1d20f646a951-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:57:55.917977 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.917974 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/822c7304-5d93-443d-aa03-1d20f646a951-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:57:55.917977 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.917984 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbx6c\" (UniqueName: \"kubernetes.io/projected/822c7304-5d93-443d-aa03-1d20f646a951-kube-api-access-vbx6c\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 17:57:55.946618 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.946589 2582 generic.go:356] "Generic (PLEG): container finished" podID="822c7304-5d93-443d-aa03-1d20f646a951" containerID="a4af6de8f18f763ef8dc09186a4dcd71752a3fdc580c79a8c644d6382fb3f987" exitCode=0 Jul 16 17:57:55.946958 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.946637 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" Jul 16 17:57:55.946958 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.946659 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" event={"ID":"822c7304-5d93-443d-aa03-1d20f646a951","Type":"ContainerDied","Data":"a4af6de8f18f763ef8dc09186a4dcd71752a3fdc580c79a8c644d6382fb3f987"} Jul 16 17:57:55.946958 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.946723 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv" event={"ID":"822c7304-5d93-443d-aa03-1d20f646a951","Type":"ContainerDied","Data":"b03a2845f03327bcea6f283c60ad68d89816bcd9ac88d87bb0bc24a5e361af38"} Jul 16 17:57:55.946958 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.946762 2582 scope.go:117] "RemoveContainer" containerID="660399f8045b107196b455e25753c0eb1b34582f9398b97e9cb2483a3a499c48" Jul 16 17:57:55.948316 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.948287 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" event={"ID":"05e1f5e2-49a6-4db2-94df-a43dd7e60837","Type":"ContainerDied","Data":"a99fade8c5b21f17ad779b1d1f2dc3c32a9720183b015c7e233c1eb6c620c408"} Jul 16 17:57:55.948407 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.948332 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl" Jul 16 17:57:55.956352 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.956336 2582 scope.go:117] "RemoveContainer" containerID="a4af6de8f18f763ef8dc09186a4dcd71752a3fdc580c79a8c644d6382fb3f987" Jul 16 17:57:55.964351 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.964332 2582 scope.go:117] "RemoveContainer" containerID="660399f8045b107196b455e25753c0eb1b34582f9398b97e9cb2483a3a499c48" Jul 16 17:57:55.964611 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:57:55.964594 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660399f8045b107196b455e25753c0eb1b34582f9398b97e9cb2483a3a499c48\": container with ID starting with 660399f8045b107196b455e25753c0eb1b34582f9398b97e9cb2483a3a499c48 not found: ID does not exist" containerID="660399f8045b107196b455e25753c0eb1b34582f9398b97e9cb2483a3a499c48" Jul 16 17:57:55.964660 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.964621 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660399f8045b107196b455e25753c0eb1b34582f9398b97e9cb2483a3a499c48"} err="failed to get container status \"660399f8045b107196b455e25753c0eb1b34582f9398b97e9cb2483a3a499c48\": rpc error: code = NotFound desc = could not find container \"660399f8045b107196b455e25753c0eb1b34582f9398b97e9cb2483a3a499c48\": container with ID starting with 660399f8045b107196b455e25753c0eb1b34582f9398b97e9cb2483a3a499c48 not found: ID does not exist" Jul 16 17:57:55.964660 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.964635 2582 scope.go:117] "RemoveContainer" containerID="a4af6de8f18f763ef8dc09186a4dcd71752a3fdc580c79a8c644d6382fb3f987" Jul 16 17:57:55.964900 ip-10-0-139-103 kubenswrapper[2582]: E0716 17:57:55.964885 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4af6de8f18f763ef8dc09186a4dcd71752a3fdc580c79a8c644d6382fb3f987\": container with ID starting with a4af6de8f18f763ef8dc09186a4dcd71752a3fdc580c79a8c644d6382fb3f987 not found: ID does not exist" containerID="a4af6de8f18f763ef8dc09186a4dcd71752a3fdc580c79a8c644d6382fb3f987" Jul 16 17:57:55.964943 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.964904 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4af6de8f18f763ef8dc09186a4dcd71752a3fdc580c79a8c644d6382fb3f987"} err="failed to get container status \"a4af6de8f18f763ef8dc09186a4dcd71752a3fdc580c79a8c644d6382fb3f987\": rpc error: code = NotFound desc = could not find container \"a4af6de8f18f763ef8dc09186a4dcd71752a3fdc580c79a8c644d6382fb3f987\": container with ID starting with a4af6de8f18f763ef8dc09186a4dcd71752a3fdc580c79a8c644d6382fb3f987 not found: ID does not exist" Jul 16 17:57:55.964943 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.964915 2582 scope.go:117] "RemoveContainer" containerID="a0e94790de26d695114b9c42e4f07632654aa1e4b93afb759407a1d50e072a1d" Jul 16 17:57:55.969096 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.969076 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv"] Jul 16 17:57:55.972608 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.972592 2582 scope.go:117] "RemoveContainer" containerID="af79412c85cf4fc455e06c66ed572e27ae1ea521e7811c6ca0a8159935fc2cf6" Jul 16 17:57:55.973616 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.973598 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6687f-predictor-547df57596-fmjkv"] Jul 16 17:57:55.985514 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.985493 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl"] Jul 16 17:57:55.988344 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:55.988323 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6687f-predictor-769fddcb6f-tfzgl"] Jul 16 17:57:56.586156 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:56.586111 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" path="/var/lib/kubelet/pods/05e1f5e2-49a6-4db2-94df-a43dd7e60837/volumes" Jul 16 17:57:56.586559 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:56.586543 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="822c7304-5d93-443d-aa03-1d20f646a951" path="/var/lib/kubelet/pods/822c7304-5d93-443d-aa03-1d20f646a951/volumes" Jul 16 17:57:58.941572 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:58.941538 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:57:58.942165 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:58.942138 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 16 17:57:59.762596 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:59.762565 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 17:57:59.944934 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:59.944904 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:57:59.945546 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:57:59.945512 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 16 17:58:00.774648 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:58:00.774570 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 17:58:08.943047 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:58:08.943008 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 16 17:58:09.945717 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:58:09.945677 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 16 17:58:18.942945 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:58:18.942903 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 16 17:58:19.945691 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:58:19.945654 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 16 17:58:28.942890 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:58:28.942848 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 16 17:58:29.945449 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:58:29.945410 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 16 17:58:38.942644 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:58:38.942612 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 17:58:39.946033 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:58:39.946004 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 17:59:10.487066 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:59:10.487036 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 17:59:10.490180 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:59:10.490159 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 17:59:10.492770 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:59:10.492748 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 17:59:10.495547 ip-10-0-139-103 kubenswrapper[2582]: I0716 17:59:10.495531 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 18:04:10.509771 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:04:10.509744 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 18:04:10.515589 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:04:10.515564 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 18:04:10.515764 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:04:10.515743 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 18:04:10.521433 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:04:10.521417 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 18:06:27.101289 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.101250 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg"] Jul 16 18:06:27.103912 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.101557 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kserve-container" containerID="cri-o://1fc5d421834cb00771549beb7dc5232c36b71546f5f4d678851215e0f82555ce" gracePeriod=30 Jul 16 18:06:27.103912 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.101575 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kube-rbac-proxy" containerID="cri-o://f023cb81c5eaf70d3561f09fdd9eac41e6ba7a800782e9eb34cc7d4b44ab330e" gracePeriod=30 Jul 16 18:06:27.154450 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.154421 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g"] Jul 16 18:06:27.154726 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.154680 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kserve-container" containerID="cri-o://d0e4bc5e29093d6679ce8b4697a91a16e4f0e12d4442935094220842d6c007d2" gracePeriod=30 Jul 16 18:06:27.154857 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.154712 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kube-rbac-proxy" containerID="cri-o://1c2907ba69ad6d07581c7f3df5aa81885ed2d8e3c216d464b8e246d19b2d9335" gracePeriod=30 Jul 16 18:06:27.219127 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.219096 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm"] Jul 16 18:06:27.219496 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.219462 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kube-rbac-proxy" Jul 16 18:06:27.219590 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.219499 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kube-rbac-proxy" Jul 16 18:06:27.219590 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.219512 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kube-rbac-proxy" Jul 16 18:06:27.219590 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.219519 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kube-rbac-proxy" Jul 16 18:06:27.219590 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.219546 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kserve-container" Jul 16 18:06:27.219590 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.219556 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kserve-container" Jul 16 18:06:27.219590 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.219579 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kserve-container" Jul 16 18:06:27.219590 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.219587 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kserve-container" Jul 16 18:06:27.219917 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.219666 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kube-rbac-proxy" Jul 16 18:06:27.219917 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.219680 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e1f5e2-49a6-4db2-94df-a43dd7e60837" containerName="kserve-container" Jul 16 18:06:27.219917 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.219689 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kserve-container" Jul 16 18:06:27.219917 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.219701 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="822c7304-5d93-443d-aa03-1d20f646a951" containerName="kube-rbac-proxy" Jul 16 18:06:27.224373 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.224352 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:27.227415 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.227389 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a019b-predictor-serving-cert\"" Jul 16 18:06:27.227877 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.227857 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a019b-kube-rbac-proxy-sar-config\"" Jul 16 18:06:27.240640 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.240414 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm"] Jul 16 18:06:27.280995 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.280959 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tktz\" (UniqueName: \"kubernetes.io/projected/02e68e74-6ad3-43ac-bd1d-dabe89c07510-kube-api-access-5tktz\") pod \"success-200-isvc-a019b-predictor-54d5fc55df-ccngm\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:27.281135 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.281012 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e68e74-6ad3-43ac-bd1d-dabe89c07510-proxy-tls\") pod \"success-200-isvc-a019b-predictor-54d5fc55df-ccngm\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:27.281135 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.281119 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e68e74-6ad3-43ac-bd1d-dabe89c07510-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a019b-predictor-54d5fc55df-ccngm\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:27.311971 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.311939 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld"] Jul 16 18:06:27.315073 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.315055 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:27.318405 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.318377 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a019b-predictor-serving-cert\"" Jul 16 18:06:27.324683 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.324662 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a019b-kube-rbac-proxy-sar-config\"" Jul 16 18:06:27.334044 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.334022 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld"] Jul 16 18:06:27.382205 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.382128 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f53a82-a7e5-4703-9241-d93ff28b09c6-proxy-tls\") pod \"error-404-isvc-a019b-predictor-5578757556-bfcld\" (UID: \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\") " pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:27.382205 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.382179 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tktz\" (UniqueName: \"kubernetes.io/projected/02e68e74-6ad3-43ac-bd1d-dabe89c07510-kube-api-access-5tktz\") pod \"success-200-isvc-a019b-predictor-54d5fc55df-ccngm\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:27.382408 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.382207 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc57g\" (UniqueName: \"kubernetes.io/projected/d8f53a82-a7e5-4703-9241-d93ff28b09c6-kube-api-access-mc57g\") pod \"error-404-isvc-a019b-predictor-5578757556-bfcld\" (UID: \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\") " pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:27.382408 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.382237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e68e74-6ad3-43ac-bd1d-dabe89c07510-proxy-tls\") pod \"success-200-isvc-a019b-predictor-54d5fc55df-ccngm\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:27.382408 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.382280 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e68e74-6ad3-43ac-bd1d-dabe89c07510-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a019b-predictor-54d5fc55df-ccngm\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:27.382408 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.382320 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f53a82-a7e5-4703-9241-d93ff28b09c6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a019b-predictor-5578757556-bfcld\" (UID: \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\") " pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:27.382408 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:06:27.382353 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a019b-predictor-serving-cert: secret "success-200-isvc-a019b-predictor-serving-cert" not found Jul 16 18:06:27.382654 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:06:27.382417 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e68e74-6ad3-43ac-bd1d-dabe89c07510-proxy-tls podName:02e68e74-6ad3-43ac-bd1d-dabe89c07510 nodeName:}" failed. No retries permitted until 2026-07-16 18:06:27.882400421 +0000 UTC m=+1937.899629741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/02e68e74-6ad3-43ac-bd1d-dabe89c07510-proxy-tls") pod "success-200-isvc-a019b-predictor-54d5fc55df-ccngm" (UID: "02e68e74-6ad3-43ac-bd1d-dabe89c07510") : secret "success-200-isvc-a019b-predictor-serving-cert" not found Jul 16 18:06:27.383190 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.383165 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e68e74-6ad3-43ac-bd1d-dabe89c07510-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a019b-predictor-54d5fc55df-ccngm\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:27.391270 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.391241 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tktz\" (UniqueName: \"kubernetes.io/projected/02e68e74-6ad3-43ac-bd1d-dabe89c07510-kube-api-access-5tktz\") pod \"success-200-isvc-a019b-predictor-54d5fc55df-ccngm\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:27.483489 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.483429 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f53a82-a7e5-4703-9241-d93ff28b09c6-proxy-tls\") pod \"error-404-isvc-a019b-predictor-5578757556-bfcld\" (UID: \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\") " pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:27.483659 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.483536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc57g\" (UniqueName: \"kubernetes.io/projected/d8f53a82-a7e5-4703-9241-d93ff28b09c6-kube-api-access-mc57g\") pod \"error-404-isvc-a019b-predictor-5578757556-bfcld\" (UID: \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\") " pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:27.483659 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:06:27.483571 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a019b-predictor-serving-cert: secret "error-404-isvc-a019b-predictor-serving-cert" not found Jul 16 18:06:27.483659 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.483619 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f53a82-a7e5-4703-9241-d93ff28b09c6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a019b-predictor-5578757556-bfcld\" (UID: \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\") " pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:27.483659 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:06:27.483641 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8f53a82-a7e5-4703-9241-d93ff28b09c6-proxy-tls podName:d8f53a82-a7e5-4703-9241-d93ff28b09c6 nodeName:}" failed. No retries permitted until 2026-07-16 18:06:27.983626271 +0000 UTC m=+1938.000855591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d8f53a82-a7e5-4703-9241-d93ff28b09c6-proxy-tls") pod "error-404-isvc-a019b-predictor-5578757556-bfcld" (UID: "d8f53a82-a7e5-4703-9241-d93ff28b09c6") : secret "error-404-isvc-a019b-predictor-serving-cert" not found Jul 16 18:06:27.484274 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.484253 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f53a82-a7e5-4703-9241-d93ff28b09c6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a019b-predictor-5578757556-bfcld\" (UID: \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\") " pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:27.492193 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.492171 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc57g\" (UniqueName: \"kubernetes.io/projected/d8f53a82-a7e5-4703-9241-d93ff28b09c6-kube-api-access-mc57g\") pod \"error-404-isvc-a019b-predictor-5578757556-bfcld\" (UID: \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\") " pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:27.887456 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.887417 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e68e74-6ad3-43ac-bd1d-dabe89c07510-proxy-tls\") pod \"success-200-isvc-a019b-predictor-54d5fc55df-ccngm\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:27.887636 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:06:27.887572 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a019b-predictor-serving-cert: secret "success-200-isvc-a019b-predictor-serving-cert" not found Jul 16 18:06:27.887636 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:06:27.887630 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e68e74-6ad3-43ac-bd1d-dabe89c07510-proxy-tls podName:02e68e74-6ad3-43ac-bd1d-dabe89c07510 nodeName:}" failed. No retries permitted until 2026-07-16 18:06:28.887615844 +0000 UTC m=+1938.904845164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/02e68e74-6ad3-43ac-bd1d-dabe89c07510-proxy-tls") pod "success-200-isvc-a019b-predictor-54d5fc55df-ccngm" (UID: "02e68e74-6ad3-43ac-bd1d-dabe89c07510") : secret "success-200-isvc-a019b-predictor-serving-cert" not found Jul 16 18:06:27.902054 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.902027 2582 generic.go:356] "Generic (PLEG): container finished" podID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerID="1c2907ba69ad6d07581c7f3df5aa81885ed2d8e3c216d464b8e246d19b2d9335" exitCode=2 Jul 16 18:06:27.902195 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.902111 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" event={"ID":"704b81b9-c1f6-45c8-9b9e-406696fcb9b5","Type":"ContainerDied","Data":"1c2907ba69ad6d07581c7f3df5aa81885ed2d8e3c216d464b8e246d19b2d9335"} Jul 16 18:06:27.903785 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.903763 2582 generic.go:356] "Generic (PLEG): container finished" podID="95720232-9c66-4c33-9431-65122d0214fa" containerID="f023cb81c5eaf70d3561f09fdd9eac41e6ba7a800782e9eb34cc7d4b44ab330e" exitCode=2 Jul 16 18:06:27.903858 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.903793 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" event={"ID":"95720232-9c66-4c33-9431-65122d0214fa","Type":"ContainerDied","Data":"f023cb81c5eaf70d3561f09fdd9eac41e6ba7a800782e9eb34cc7d4b44ab330e"} Jul 16 18:06:27.988769 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.988744 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f53a82-a7e5-4703-9241-d93ff28b09c6-proxy-tls\") pod \"error-404-isvc-a019b-predictor-5578757556-bfcld\" (UID: \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\") " pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:27.991211 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:27.991192 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f53a82-a7e5-4703-9241-d93ff28b09c6-proxy-tls\") pod \"error-404-isvc-a019b-predictor-5578757556-bfcld\" (UID: \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\") " pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:28.225197 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:28.225107 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:28.355049 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:28.355027 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld"] Jul 16 18:06:28.357183 ip-10-0-139-103 kubenswrapper[2582]: W0716 18:06:28.357152 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f53a82_a7e5_4703_9241_d93ff28b09c6.slice/crio-eea009037f3a3e84e88d4c0261deed32169bf3cc1dd3ccea06356a1f025c2f34 WatchSource:0}: Error finding container eea009037f3a3e84e88d4c0261deed32169bf3cc1dd3ccea06356a1f025c2f34: Status 404 returned error can't find the container with id eea009037f3a3e84e88d4c0261deed32169bf3cc1dd3ccea06356a1f025c2f34 Jul 16 18:06:28.897501 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:28.897455 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e68e74-6ad3-43ac-bd1d-dabe89c07510-proxy-tls\") pod \"success-200-isvc-a019b-predictor-54d5fc55df-ccngm\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:28.900038 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:28.900016 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e68e74-6ad3-43ac-bd1d-dabe89c07510-proxy-tls\") pod \"success-200-isvc-a019b-predictor-54d5fc55df-ccngm\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:28.909908 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:28.909883 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" event={"ID":"d8f53a82-a7e5-4703-9241-d93ff28b09c6","Type":"ContainerStarted","Data":"63a4bcc0d748e56dab8b7dbb0f7fa0b399070b6b08f46abf3b2c15f1d5f3a269"} Jul 16 18:06:28.910015 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:28.909940 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" event={"ID":"d8f53a82-a7e5-4703-9241-d93ff28b09c6","Type":"ContainerStarted","Data":"84de2af262c92efa9b26c1715da18eb73b3c85f2e23ebd4581aa993de896090a"} Jul 16 18:06:28.910015 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:28.909971 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:28.910015 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:28.909989 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" event={"ID":"d8f53a82-a7e5-4703-9241-d93ff28b09c6","Type":"ContainerStarted","Data":"eea009037f3a3e84e88d4c0261deed32169bf3cc1dd3ccea06356a1f025c2f34"} Jul 16 18:06:28.928063 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:28.928012 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" podStartSLOduration=1.927999347 podStartE2EDuration="1.927999347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:06:28.926377346 +0000 UTC m=+1938.943606704" watchObservedRunningTime="2026-07-16 18:06:28.927999347 +0000 UTC m=+1938.945228688" Jul 16 18:06:29.037748 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:29.037724 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:29.167419 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:29.167394 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm"] Jul 16 18:06:29.169733 ip-10-0-139-103 kubenswrapper[2582]: W0716 18:06:29.169703 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e68e74_6ad3_43ac_bd1d_dabe89c07510.slice/crio-e24653c7c9e96594af34c9cee0ba1e1cc8774d4cc9e579f76af180df05c52c4a WatchSource:0}: Error finding container e24653c7c9e96594af34c9cee0ba1e1cc8774d4cc9e579f76af180df05c52c4a: Status 404 returned error can't find the container with id e24653c7c9e96594af34c9cee0ba1e1cc8774d4cc9e579f76af180df05c52c4a Jul 16 18:06:29.756722 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:29.756678 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused" Jul 16 18:06:29.761877 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:29.761849 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 16 18:06:29.915463 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:29.915424 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" event={"ID":"02e68e74-6ad3-43ac-bd1d-dabe89c07510","Type":"ContainerStarted","Data":"d7c219f862b8fe962ceacb21ae3bb3428353d6d89b28520b56996430158b2035"} Jul 16 18:06:29.915659 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:29.915512 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" event={"ID":"02e68e74-6ad3-43ac-bd1d-dabe89c07510","Type":"ContainerStarted","Data":"9f5ae837da361e8ff0d862efbc56f176e19cbb11f7993049eb3036544a2e07a6"} Jul 16 18:06:29.915659 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:29.915539 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" event={"ID":"02e68e74-6ad3-43ac-bd1d-dabe89c07510","Type":"ContainerStarted","Data":"e24653c7c9e96594af34c9cee0ba1e1cc8774d4cc9e579f76af180df05c52c4a"} Jul 16 18:06:29.915770 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:29.915729 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:29.916832 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:29.916805 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 16 18:06:29.933237 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:29.933192 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" podStartSLOduration=2.933178419 podStartE2EDuration="2.933178419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:06:29.931980735 +0000 UTC m=+1939.949210077" watchObservedRunningTime="2026-07-16 18:06:29.933178419 +0000 UTC m=+1939.950407761" Jul 16 18:06:30.150448 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.150423 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 18:06:30.209181 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.209157 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb5qq\" (UniqueName: \"kubernetes.io/projected/95720232-9c66-4c33-9431-65122d0214fa-kube-api-access-tb5qq\") pod \"95720232-9c66-4c33-9431-65122d0214fa\" (UID: \"95720232-9c66-4c33-9431-65122d0214fa\") " Jul 16 18:06:30.209318 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.209221 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95720232-9c66-4c33-9431-65122d0214fa-kube-rbac-proxy-sar-config\") pod \"95720232-9c66-4c33-9431-65122d0214fa\" (UID: \"95720232-9c66-4c33-9431-65122d0214fa\") " Jul 16 18:06:30.209359 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.209330 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95720232-9c66-4c33-9431-65122d0214fa-proxy-tls\") pod \"95720232-9c66-4c33-9431-65122d0214fa\" (UID: \"95720232-9c66-4c33-9431-65122d0214fa\") " Jul 16 18:06:30.209589 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.209567 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95720232-9c66-4c33-9431-65122d0214fa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "95720232-9c66-4c33-9431-65122d0214fa" (UID: "95720232-9c66-4c33-9431-65122d0214fa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:06:30.211411 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.211389 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95720232-9c66-4c33-9431-65122d0214fa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "95720232-9c66-4c33-9431-65122d0214fa" (UID: "95720232-9c66-4c33-9431-65122d0214fa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:06:30.211573 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.211438 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95720232-9c66-4c33-9431-65122d0214fa-kube-api-access-tb5qq" (OuterVolumeSpecName: "kube-api-access-tb5qq") pod "95720232-9c66-4c33-9431-65122d0214fa" (UID: "95720232-9c66-4c33-9431-65122d0214fa"). InnerVolumeSpecName "kube-api-access-tb5qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:06:30.309909 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.309847 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb5qq\" (UniqueName: \"kubernetes.io/projected/95720232-9c66-4c33-9431-65122d0214fa-kube-api-access-tb5qq\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:06:30.309909 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.309878 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95720232-9c66-4c33-9431-65122d0214fa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:06:30.309909 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.309889 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95720232-9c66-4c33-9431-65122d0214fa-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:06:30.769045 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.769003 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused" Jul 16 18:06:30.774311 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.774289 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Jul 16 18:06:30.920533 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.920441 2582 generic.go:356] "Generic (PLEG): container finished" podID="95720232-9c66-4c33-9431-65122d0214fa" containerID="1fc5d421834cb00771549beb7dc5232c36b71546f5f4d678851215e0f82555ce" exitCode=0 Jul 16 18:06:30.920533 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.920516 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" Jul 16 18:06:30.920533 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.920525 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" event={"ID":"95720232-9c66-4c33-9431-65122d0214fa","Type":"ContainerDied","Data":"1fc5d421834cb00771549beb7dc5232c36b71546f5f4d678851215e0f82555ce"} Jul 16 18:06:30.920743 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.920588 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg" event={"ID":"95720232-9c66-4c33-9431-65122d0214fa","Type":"ContainerDied","Data":"aa0f23c8fea959deeea6e8d8a14dc0c719cfb19ccfbd52747323e64979ef40d9"} Jul 16 18:06:30.920743 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.920626 2582 scope.go:117] "RemoveContainer" containerID="f023cb81c5eaf70d3561f09fdd9eac41e6ba7a800782e9eb34cc7d4b44ab330e" Jul 16 18:06:30.921234 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.921207 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 16 18:06:30.922032 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.921923 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:30.922158 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.922101 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:30.922493 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.922440 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Jul 16 18:06:30.932878 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.932861 2582 scope.go:117] "RemoveContainer" containerID="1fc5d421834cb00771549beb7dc5232c36b71546f5f4d678851215e0f82555ce" Jul 16 18:06:30.939912 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.939887 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg"] Jul 16 18:06:30.940882 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.940866 2582 scope.go:117] "RemoveContainer" containerID="f023cb81c5eaf70d3561f09fdd9eac41e6ba7a800782e9eb34cc7d4b44ab330e" Jul 16 18:06:30.941157 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:06:30.941139 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f023cb81c5eaf70d3561f09fdd9eac41e6ba7a800782e9eb34cc7d4b44ab330e\": container with ID starting with f023cb81c5eaf70d3561f09fdd9eac41e6ba7a800782e9eb34cc7d4b44ab330e not found: ID does not exist" containerID="f023cb81c5eaf70d3561f09fdd9eac41e6ba7a800782e9eb34cc7d4b44ab330e" Jul 16 18:06:30.941200 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.941165 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f023cb81c5eaf70d3561f09fdd9eac41e6ba7a800782e9eb34cc7d4b44ab330e"} err="failed to get container status \"f023cb81c5eaf70d3561f09fdd9eac41e6ba7a800782e9eb34cc7d4b44ab330e\": rpc error: code = NotFound desc = could not find container \"f023cb81c5eaf70d3561f09fdd9eac41e6ba7a800782e9eb34cc7d4b44ab330e\": container with ID starting with f023cb81c5eaf70d3561f09fdd9eac41e6ba7a800782e9eb34cc7d4b44ab330e not found: ID does not exist" Jul 16 18:06:30.941200 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.941180 2582 scope.go:117] "RemoveContainer" containerID="1fc5d421834cb00771549beb7dc5232c36b71546f5f4d678851215e0f82555ce" Jul 16 18:06:30.941421 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:06:30.941403 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc5d421834cb00771549beb7dc5232c36b71546f5f4d678851215e0f82555ce\": container with ID starting with 1fc5d421834cb00771549beb7dc5232c36b71546f5f4d678851215e0f82555ce not found: ID does not exist" containerID="1fc5d421834cb00771549beb7dc5232c36b71546f5f4d678851215e0f82555ce" Jul 16 18:06:30.941527 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.941422 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc5d421834cb00771549beb7dc5232c36b71546f5f4d678851215e0f82555ce"} err="failed to get container status \"1fc5d421834cb00771549beb7dc5232c36b71546f5f4d678851215e0f82555ce\": rpc error: code = NotFound desc = could not find container \"1fc5d421834cb00771549beb7dc5232c36b71546f5f4d678851215e0f82555ce\": container with ID starting with 1fc5d421834cb00771549beb7dc5232c36b71546f5f4d678851215e0f82555ce not found: ID does not exist" Jul 16 18:06:30.941527 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:30.941450 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-efe2d-predictor-564c8f4947-m77mg"] Jul 16 18:06:31.925925 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:31.925887 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Jul 16 18:06:32.586199 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:32.586158 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95720232-9c66-4c33-9431-65122d0214fa" path="/var/lib/kubelet/pods/95720232-9c66-4c33-9431-65122d0214fa/volumes" Jul 16 18:06:33.294258 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.294235 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 18:06:33.432786 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.432713 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-kube-rbac-proxy-sar-config\") pod \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\" (UID: \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\") " Jul 16 18:06:33.432786 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.432776 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-proxy-tls\") pod \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\" (UID: \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\") " Jul 16 18:06:33.432996 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.432809 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxwdk\" (UniqueName: \"kubernetes.io/projected/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-kube-api-access-bxwdk\") pod \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\" (UID: \"704b81b9-c1f6-45c8-9b9e-406696fcb9b5\") " Jul 16 18:06:33.433140 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.433117 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "704b81b9-c1f6-45c8-9b9e-406696fcb9b5" (UID: "704b81b9-c1f6-45c8-9b9e-406696fcb9b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:06:33.435069 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.435045 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "704b81b9-c1f6-45c8-9b9e-406696fcb9b5" (UID: "704b81b9-c1f6-45c8-9b9e-406696fcb9b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:06:33.435069 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.435056 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-kube-api-access-bxwdk" (OuterVolumeSpecName: "kube-api-access-bxwdk") pod "704b81b9-c1f6-45c8-9b9e-406696fcb9b5" (UID: "704b81b9-c1f6-45c8-9b9e-406696fcb9b5"). InnerVolumeSpecName "kube-api-access-bxwdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:06:33.533970 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.533939 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxwdk\" (UniqueName: \"kubernetes.io/projected/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-kube-api-access-bxwdk\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:06:33.533970 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.533966 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:06:33.534100 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.533982 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704b81b9-c1f6-45c8-9b9e-406696fcb9b5-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:06:33.934849 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.934821 2582 generic.go:356] "Generic (PLEG): container finished" podID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerID="d0e4bc5e29093d6679ce8b4697a91a16e4f0e12d4442935094220842d6c007d2" exitCode=0 Jul 16 18:06:33.935003 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.934871 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" Jul 16 18:06:33.935003 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.934873 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" event={"ID":"704b81b9-c1f6-45c8-9b9e-406696fcb9b5","Type":"ContainerDied","Data":"d0e4bc5e29093d6679ce8b4697a91a16e4f0e12d4442935094220842d6c007d2"} Jul 16 18:06:33.935003 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.934972 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g" event={"ID":"704b81b9-c1f6-45c8-9b9e-406696fcb9b5","Type":"ContainerDied","Data":"51fc0aa025e0b201a9c75e92a297f88d2f84faa8a4e92d19f97c505ea7142f65"} Jul 16 18:06:33.935111 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.935004 2582 scope.go:117] "RemoveContainer" containerID="1c2907ba69ad6d07581c7f3df5aa81885ed2d8e3c216d464b8e246d19b2d9335" Jul 16 18:06:33.943414 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.943394 2582 scope.go:117] "RemoveContainer" containerID="d0e4bc5e29093d6679ce8b4697a91a16e4f0e12d4442935094220842d6c007d2" Jul 16 18:06:33.950673 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.950651 2582 scope.go:117] "RemoveContainer" containerID="1c2907ba69ad6d07581c7f3df5aa81885ed2d8e3c216d464b8e246d19b2d9335" Jul 16 18:06:33.950944 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:06:33.950923 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2907ba69ad6d07581c7f3df5aa81885ed2d8e3c216d464b8e246d19b2d9335\": container with ID starting with 1c2907ba69ad6d07581c7f3df5aa81885ed2d8e3c216d464b8e246d19b2d9335 not found: ID does not exist" containerID="1c2907ba69ad6d07581c7f3df5aa81885ed2d8e3c216d464b8e246d19b2d9335" Jul 16 18:06:33.951001 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.950951 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2907ba69ad6d07581c7f3df5aa81885ed2d8e3c216d464b8e246d19b2d9335"} err="failed to get container status \"1c2907ba69ad6d07581c7f3df5aa81885ed2d8e3c216d464b8e246d19b2d9335\": rpc error: code = NotFound desc = could not find container \"1c2907ba69ad6d07581c7f3df5aa81885ed2d8e3c216d464b8e246d19b2d9335\": container with ID starting with 1c2907ba69ad6d07581c7f3df5aa81885ed2d8e3c216d464b8e246d19b2d9335 not found: ID does not exist" Jul 16 18:06:33.951001 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.950970 2582 scope.go:117] "RemoveContainer" containerID="d0e4bc5e29093d6679ce8b4697a91a16e4f0e12d4442935094220842d6c007d2" Jul 16 18:06:33.951209 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:06:33.951192 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e4bc5e29093d6679ce8b4697a91a16e4f0e12d4442935094220842d6c007d2\": container with ID starting with d0e4bc5e29093d6679ce8b4697a91a16e4f0e12d4442935094220842d6c007d2 not found: ID does not exist" containerID="d0e4bc5e29093d6679ce8b4697a91a16e4f0e12d4442935094220842d6c007d2" Jul 16 18:06:33.951252 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.951215 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e4bc5e29093d6679ce8b4697a91a16e4f0e12d4442935094220842d6c007d2"} err="failed to get container status \"d0e4bc5e29093d6679ce8b4697a91a16e4f0e12d4442935094220842d6c007d2\": rpc error: code = NotFound desc = could not find container \"d0e4bc5e29093d6679ce8b4697a91a16e4f0e12d4442935094220842d6c007d2\": container with ID starting with d0e4bc5e29093d6679ce8b4697a91a16e4f0e12d4442935094220842d6c007d2 not found: ID does not exist" Jul 16 18:06:33.955981 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.955957 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g"] Jul 16 18:06:33.962463 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:33.962443 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-efe2d-predictor-568fd6d6fd-br76g"] Jul 16 18:06:34.584962 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:34.584922 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" path="/var/lib/kubelet/pods/704b81b9-c1f6-45c8-9b9e-406696fcb9b5/volumes" Jul 16 18:06:35.925669 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:35.925644 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:06:35.926144 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:35.926120 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 16 18:06:36.931070 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:36.931040 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:06:36.931464 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:36.931438 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Jul 16 18:06:45.926974 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:45.926939 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 16 18:06:46.931870 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:46.931830 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Jul 16 18:06:55.926920 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:55.926876 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 16 18:06:56.931784 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:06:56.931742 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Jul 16 18:07:05.926742 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:05.926658 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 16 18:07:06.651998 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.651965 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb"] Jul 16 18:07:06.652207 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.652168 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kserve-container" containerID="cri-o://6af61e574e20b7428a0bd93f1c06b9b8265144c16e26dc31c1db2a498beaf116" gracePeriod=30 Jul 16 18:07:06.652340 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.652228 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kube-rbac-proxy" containerID="cri-o://b6d9ed67ed4bf44bb46418bd953b5a327115b45cae610fd73bfca408cf6c40ce" gracePeriod=30 Jul 16 18:07:06.709299 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.709214 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft"] Jul 16 18:07:06.709565 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.709510 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kserve-container" containerID="cri-o://b2fc8acb10c85af8bcaf7ed1244ad399a7f3d8cd5f9c55b827876f711eb8067c" gracePeriod=30 Jul 16 18:07:06.709697 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.709559 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kube-rbac-proxy" containerID="cri-o://62d907ad1985c73f31ca0ec56e82d1d41356fb49956f4c637ce4acd9c332daaa" gracePeriod=30 Jul 16 18:07:06.769051 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.769025 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw"] Jul 16 18:07:06.769347 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.769334 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kserve-container" Jul 16 18:07:06.769389 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.769350 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kserve-container" Jul 16 18:07:06.769389 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.769364 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kube-rbac-proxy" Jul 16 18:07:06.769389 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.769370 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kube-rbac-proxy" Jul 16 18:07:06.769389 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.769381 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kserve-container" Jul 16 18:07:06.769389 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.769386 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kserve-container" Jul 16 18:07:06.769563 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.769398 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kube-rbac-proxy" Jul 16 18:07:06.769563 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.769404 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kube-rbac-proxy" Jul 16 18:07:06.769563 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.769449 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kube-rbac-proxy" Jul 16 18:07:06.769563 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.769458 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kube-rbac-proxy" Jul 16 18:07:06.769563 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.769464 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="95720232-9c66-4c33-9431-65122d0214fa" containerName="kserve-container" Jul 16 18:07:06.769563 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.769497 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="704b81b9-c1f6-45c8-9b9e-406696fcb9b5" containerName="kserve-container" Jul 16 18:07:06.772462 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.772445 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:06.774644 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.774626 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-97269-predictor-serving-cert\"" Jul 16 18:07:06.774871 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.774851 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-97269-kube-rbac-proxy-sar-config\"" Jul 16 18:07:06.782688 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.782660 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw"] Jul 16 18:07:06.869001 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.868969 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g"] Jul 16 18:07:06.872246 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.872230 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:06.874619 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.874598 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-97269-kube-rbac-proxy-sar-config\"" Jul 16 18:07:06.874722 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.874636 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-97269-predictor-serving-cert\"" Jul 16 18:07:06.883111 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.882979 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g"] Jul 16 18:07:06.897167 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.897133 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef32f086-22fa-4544-a6e3-b6e27db85bfc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-97269-predictor-b796c5fd5-tzshw\" (UID: \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\") " pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:06.897264 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.897170 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cb9m\" (UniqueName: \"kubernetes.io/projected/ef32f086-22fa-4544-a6e3-b6e27db85bfc-kube-api-access-6cb9m\") pod \"success-200-isvc-97269-predictor-b796c5fd5-tzshw\" (UID: \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\") " pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:06.897264 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.897205 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae46f104-584c-44ce-ae08-507d364f8971-proxy-tls\") pod \"error-404-isvc-97269-predictor-7d7f6f997b-9t55g\" (UID: \"ae46f104-584c-44ce-ae08-507d364f8971\") " pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:06.897351 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.897285 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef32f086-22fa-4544-a6e3-b6e27db85bfc-proxy-tls\") pod \"success-200-isvc-97269-predictor-b796c5fd5-tzshw\" (UID: \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\") " pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:06.897351 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.897325 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzv7z\" (UniqueName: \"kubernetes.io/projected/ae46f104-584c-44ce-ae08-507d364f8971-kube-api-access-rzv7z\") pod \"error-404-isvc-97269-predictor-7d7f6f997b-9t55g\" (UID: \"ae46f104-584c-44ce-ae08-507d364f8971\") " pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:06.897417 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.897357 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae46f104-584c-44ce-ae08-507d364f8971-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-97269-predictor-7d7f6f997b-9t55g\" (UID: \"ae46f104-584c-44ce-ae08-507d364f8971\") " pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:06.932070 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.931997 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Jul 16 18:07:06.998205 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.998165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzv7z\" (UniqueName: \"kubernetes.io/projected/ae46f104-584c-44ce-ae08-507d364f8971-kube-api-access-rzv7z\") pod \"error-404-isvc-97269-predictor-7d7f6f997b-9t55g\" (UID: \"ae46f104-584c-44ce-ae08-507d364f8971\") " pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:06.998377 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.998211 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae46f104-584c-44ce-ae08-507d364f8971-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-97269-predictor-7d7f6f997b-9t55g\" (UID: \"ae46f104-584c-44ce-ae08-507d364f8971\") " pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:06.998377 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.998270 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef32f086-22fa-4544-a6e3-b6e27db85bfc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-97269-predictor-b796c5fd5-tzshw\" (UID: \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\") " pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:06.998377 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.998297 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cb9m\" (UniqueName: \"kubernetes.io/projected/ef32f086-22fa-4544-a6e3-b6e27db85bfc-kube-api-access-6cb9m\") pod \"success-200-isvc-97269-predictor-b796c5fd5-tzshw\" (UID: \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\") " pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:06.998377 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.998332 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae46f104-584c-44ce-ae08-507d364f8971-proxy-tls\") pod \"error-404-isvc-97269-predictor-7d7f6f997b-9t55g\" (UID: \"ae46f104-584c-44ce-ae08-507d364f8971\") " pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:06.998377 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.998364 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef32f086-22fa-4544-a6e3-b6e27db85bfc-proxy-tls\") pod \"success-200-isvc-97269-predictor-b796c5fd5-tzshw\" (UID: \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\") " pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:06.998687 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:07:06.998668 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-97269-predictor-serving-cert: secret "error-404-isvc-97269-predictor-serving-cert" not found Jul 16 18:07:06.998769 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:07:06.998755 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae46f104-584c-44ce-ae08-507d364f8971-proxy-tls podName:ae46f104-584c-44ce-ae08-507d364f8971 nodeName:}" failed. No retries permitted until 2026-07-16 18:07:07.498736336 +0000 UTC m=+1977.515965671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae46f104-584c-44ce-ae08-507d364f8971-proxy-tls") pod "error-404-isvc-97269-predictor-7d7f6f997b-9t55g" (UID: "ae46f104-584c-44ce-ae08-507d364f8971") : secret "error-404-isvc-97269-predictor-serving-cert" not found Jul 16 18:07:06.999010 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.998985 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef32f086-22fa-4544-a6e3-b6e27db85bfc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-97269-predictor-b796c5fd5-tzshw\" (UID: \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\") " pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:06.999126 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:06.999012 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae46f104-584c-44ce-ae08-507d364f8971-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-97269-predictor-7d7f6f997b-9t55g\" (UID: \"ae46f104-584c-44ce-ae08-507d364f8971\") " pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:07.001206 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:07.001185 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef32f086-22fa-4544-a6e3-b6e27db85bfc-proxy-tls\") pod \"success-200-isvc-97269-predictor-b796c5fd5-tzshw\" (UID: \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\") " pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:07.007197 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:07.007173 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cb9m\" (UniqueName: \"kubernetes.io/projected/ef32f086-22fa-4544-a6e3-b6e27db85bfc-kube-api-access-6cb9m\") pod \"success-200-isvc-97269-predictor-b796c5fd5-tzshw\" (UID: \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\") " pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:07.007308 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:07.007198 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzv7z\" (UniqueName: \"kubernetes.io/projected/ae46f104-584c-44ce-ae08-507d364f8971-kube-api-access-rzv7z\") pod \"error-404-isvc-97269-predictor-7d7f6f997b-9t55g\" (UID: \"ae46f104-584c-44ce-ae08-507d364f8971\") " pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:07.066391 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:07.066367 2582 generic.go:356] "Generic (PLEG): container finished" podID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerID="b6d9ed67ed4bf44bb46418bd953b5a327115b45cae610fd73bfca408cf6c40ce" exitCode=2 Jul 16 18:07:07.066528 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:07.066443 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" event={"ID":"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd","Type":"ContainerDied","Data":"b6d9ed67ed4bf44bb46418bd953b5a327115b45cae610fd73bfca408cf6c40ce"} Jul 16 18:07:07.068158 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:07.068136 2582 generic.go:356] "Generic (PLEG): container finished" podID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerID="62d907ad1985c73f31ca0ec56e82d1d41356fb49956f4c637ce4acd9c332daaa" exitCode=2 Jul 16 18:07:07.068244 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:07.068186 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" event={"ID":"77d38b5b-b79d-496b-bbd5-f0baa7173598","Type":"ContainerDied","Data":"62d907ad1985c73f31ca0ec56e82d1d41356fb49956f4c637ce4acd9c332daaa"} Jul 16 18:07:07.083490 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:07.083445 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:07.216521 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:07.216407 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw"] Jul 16 18:07:07.219020 ip-10-0-139-103 kubenswrapper[2582]: W0716 18:07:07.218986 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef32f086_22fa_4544_a6e3_b6e27db85bfc.slice/crio-78decdcdfa5242866aca1a624e611cb6d4961ac1bb99b7607abf6cd1d4a698dc WatchSource:0}: Error finding container 78decdcdfa5242866aca1a624e611cb6d4961ac1bb99b7607abf6cd1d4a698dc: Status 404 returned error can't find the container with id 78decdcdfa5242866aca1a624e611cb6d4961ac1bb99b7607abf6cd1d4a698dc Jul 16 18:07:07.501332 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:07.501301 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae46f104-584c-44ce-ae08-507d364f8971-proxy-tls\") pod \"error-404-isvc-97269-predictor-7d7f6f997b-9t55g\" (UID: \"ae46f104-584c-44ce-ae08-507d364f8971\") " pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:07.503907 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:07.503885 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae46f104-584c-44ce-ae08-507d364f8971-proxy-tls\") pod \"error-404-isvc-97269-predictor-7d7f6f997b-9t55g\" (UID: \"ae46f104-584c-44ce-ae08-507d364f8971\") " pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:07.784944 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:07.784841 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:07.915728 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:07.915640 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g"] Jul 16 18:07:07.917835 ip-10-0-139-103 kubenswrapper[2582]: W0716 18:07:07.917807 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae46f104_584c_44ce_ae08_507d364f8971.slice/crio-1846d5fa7f0a067a7d61397791c6c4d013251d63c960bd3dd29110ee08255049 WatchSource:0}: Error finding container 1846d5fa7f0a067a7d61397791c6c4d013251d63c960bd3dd29110ee08255049: Status 404 returned error can't find the container with id 1846d5fa7f0a067a7d61397791c6c4d013251d63c960bd3dd29110ee08255049 Jul 16 18:07:08.074895 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.074862 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" event={"ID":"ef32f086-22fa-4544-a6e3-b6e27db85bfc","Type":"ContainerStarted","Data":"173939769c60feb06844afb0d3776af5acb10ec442086fbd822eb98b792bbe51"} Jul 16 18:07:08.075322 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.074939 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:08.075322 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.074974 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" event={"ID":"ef32f086-22fa-4544-a6e3-b6e27db85bfc","Type":"ContainerStarted","Data":"e35960aca8bde3ceace01f9d4c0a5e0a28ec79f3f091bd75c93595580d2e8bf5"} Jul 16 18:07:08.075322 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.075005 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" event={"ID":"ef32f086-22fa-4544-a6e3-b6e27db85bfc","Type":"ContainerStarted","Data":"78decdcdfa5242866aca1a624e611cb6d4961ac1bb99b7607abf6cd1d4a698dc"} Jul 16 18:07:08.075322 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.075039 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:08.076374 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.076334 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Jul 16 18:07:08.076942 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.076908 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" event={"ID":"ae46f104-584c-44ce-ae08-507d364f8971","Type":"ContainerStarted","Data":"092f13f82034a886a7356d5779b4f47ac0fbaaf6c000669b5abf1ed5e7bfb85a"} Jul 16 18:07:08.077047 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.076969 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" event={"ID":"ae46f104-584c-44ce-ae08-507d364f8971","Type":"ContainerStarted","Data":"df4c5ea879819ab52f3fb68cf925d5c6695ee72bab4365b89becb372dc044ef6"} Jul 16 18:07:08.077047 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.076998 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" event={"ID":"ae46f104-584c-44ce-ae08-507d364f8971","Type":"ContainerStarted","Data":"1846d5fa7f0a067a7d61397791c6c4d013251d63c960bd3dd29110ee08255049"} Jul 16 18:07:08.077152 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.077044 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:08.077152 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.077096 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:08.078352 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.078249 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 16 18:07:08.093388 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.093340 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" podStartSLOduration=2.093321403 podStartE2EDuration="2.093321403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:07:08.092062521 +0000 UTC m=+1978.109291864" watchObservedRunningTime="2026-07-16 18:07:08.093321403 +0000 UTC m=+1978.110550747" Jul 16 18:07:08.108668 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.108625 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" podStartSLOduration=2.108610436 podStartE2EDuration="2.108610436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:07:08.10778186 +0000 UTC m=+1978.125011201" watchObservedRunningTime="2026-07-16 18:07:08.108610436 +0000 UTC m=+1978.125839851" Jul 16 18:07:08.933673 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.933627 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused" Jul 16 18:07:08.942354 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:08.942326 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 16 18:07:09.083879 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:09.083807 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Jul 16 18:07:09.084608 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:09.084574 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 16 18:07:09.944559 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:09.944531 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 18:07:10.020579 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.020550 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxxll\" (UniqueName: \"kubernetes.io/projected/77d38b5b-b79d-496b-bbd5-f0baa7173598-kube-api-access-qxxll\") pod \"77d38b5b-b79d-496b-bbd5-f0baa7173598\" (UID: \"77d38b5b-b79d-496b-bbd5-f0baa7173598\") " Jul 16 18:07:10.020718 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.020621 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d38b5b-b79d-496b-bbd5-f0baa7173598-proxy-tls\") pod \"77d38b5b-b79d-496b-bbd5-f0baa7173598\" (UID: \"77d38b5b-b79d-496b-bbd5-f0baa7173598\") " Jul 16 18:07:10.020718 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.020645 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d38b5b-b79d-496b-bbd5-f0baa7173598-kube-rbac-proxy-sar-config\") pod \"77d38b5b-b79d-496b-bbd5-f0baa7173598\" (UID: \"77d38b5b-b79d-496b-bbd5-f0baa7173598\") " Jul 16 18:07:10.021016 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.020991 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d38b5b-b79d-496b-bbd5-f0baa7173598-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77d38b5b-b79d-496b-bbd5-f0baa7173598" (UID: "77d38b5b-b79d-496b-bbd5-f0baa7173598"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:07:10.022777 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.022749 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d38b5b-b79d-496b-bbd5-f0baa7173598-kube-api-access-qxxll" (OuterVolumeSpecName: "kube-api-access-qxxll") pod "77d38b5b-b79d-496b-bbd5-f0baa7173598" (UID: "77d38b5b-b79d-496b-bbd5-f0baa7173598"). InnerVolumeSpecName "kube-api-access-qxxll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:07:10.022849 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.022824 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d38b5b-b79d-496b-bbd5-f0baa7173598-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77d38b5b-b79d-496b-bbd5-f0baa7173598" (UID: "77d38b5b-b79d-496b-bbd5-f0baa7173598"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:07:10.090376 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.090290 2582 generic.go:356] "Generic (PLEG): container finished" podID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerID="b2fc8acb10c85af8bcaf7ed1244ad399a7f3d8cd5f9c55b827876f711eb8067c" exitCode=0 Jul 16 18:07:10.090376 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.090338 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" event={"ID":"77d38b5b-b79d-496b-bbd5-f0baa7173598","Type":"ContainerDied","Data":"b2fc8acb10c85af8bcaf7ed1244ad399a7f3d8cd5f9c55b827876f711eb8067c"} Jul 16 18:07:10.090842 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.090352 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" Jul 16 18:07:10.090842 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.090450 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft" event={"ID":"77d38b5b-b79d-496b-bbd5-f0baa7173598","Type":"ContainerDied","Data":"6bf966943b2e81f0cc2ced25b1476f925233b5a71f8f8d29926a98c19bb03491"} Jul 16 18:07:10.090842 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.090496 2582 scope.go:117] "RemoveContainer" containerID="62d907ad1985c73f31ca0ec56e82d1d41356fb49956f4c637ce4acd9c332daaa" Jul 16 18:07:10.099374 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.099355 2582 scope.go:117] "RemoveContainer" containerID="b2fc8acb10c85af8bcaf7ed1244ad399a7f3d8cd5f9c55b827876f711eb8067c" Jul 16 18:07:10.106714 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.106685 2582 scope.go:117] "RemoveContainer" containerID="62d907ad1985c73f31ca0ec56e82d1d41356fb49956f4c637ce4acd9c332daaa" Jul 16 18:07:10.106974 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:07:10.106954 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d907ad1985c73f31ca0ec56e82d1d41356fb49956f4c637ce4acd9c332daaa\": container with ID starting with 62d907ad1985c73f31ca0ec56e82d1d41356fb49956f4c637ce4acd9c332daaa not found: ID does not exist" containerID="62d907ad1985c73f31ca0ec56e82d1d41356fb49956f4c637ce4acd9c332daaa" Jul 16 18:07:10.107026 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.106983 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d907ad1985c73f31ca0ec56e82d1d41356fb49956f4c637ce4acd9c332daaa"} err="failed to get container status \"62d907ad1985c73f31ca0ec56e82d1d41356fb49956f4c637ce4acd9c332daaa\": rpc error: code = NotFound desc = could not find container \"62d907ad1985c73f31ca0ec56e82d1d41356fb49956f4c637ce4acd9c332daaa\": container with ID starting with 62d907ad1985c73f31ca0ec56e82d1d41356fb49956f4c637ce4acd9c332daaa not found: ID does not exist" Jul 16 18:07:10.107026 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.107000 2582 scope.go:117] "RemoveContainer" containerID="b2fc8acb10c85af8bcaf7ed1244ad399a7f3d8cd5f9c55b827876f711eb8067c" Jul 16 18:07:10.107248 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:07:10.107231 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2fc8acb10c85af8bcaf7ed1244ad399a7f3d8cd5f9c55b827876f711eb8067c\": container with ID starting with b2fc8acb10c85af8bcaf7ed1244ad399a7f3d8cd5f9c55b827876f711eb8067c not found: ID does not exist" containerID="b2fc8acb10c85af8bcaf7ed1244ad399a7f3d8cd5f9c55b827876f711eb8067c" Jul 16 18:07:10.107284 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.107254 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2fc8acb10c85af8bcaf7ed1244ad399a7f3d8cd5f9c55b827876f711eb8067c"} err="failed to get container status \"b2fc8acb10c85af8bcaf7ed1244ad399a7f3d8cd5f9c55b827876f711eb8067c\": rpc error: code = NotFound desc = could not find container \"b2fc8acb10c85af8bcaf7ed1244ad399a7f3d8cd5f9c55b827876f711eb8067c\": container with ID starting with b2fc8acb10c85af8bcaf7ed1244ad399a7f3d8cd5f9c55b827876f711eb8067c not found: ID does not exist" Jul 16 18:07:10.112211 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.112178 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft"] Jul 16 18:07:10.113832 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.113814 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a90bb-predictor-675fc99bf9-zwxft"] Jul 16 18:07:10.121707 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.121688 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxxll\" (UniqueName: \"kubernetes.io/projected/77d38b5b-b79d-496b-bbd5-f0baa7173598-kube-api-access-qxxll\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:07:10.121765 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.121707 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d38b5b-b79d-496b-bbd5-f0baa7173598-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:07:10.121765 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.121718 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d38b5b-b79d-496b-bbd5-f0baa7173598-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:07:10.587855 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.587814 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" path="/var/lib/kubelet/pods/77d38b5b-b79d-496b-bbd5-f0baa7173598/volumes" Jul 16 18:07:10.597879 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.597851 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 18:07:10.601403 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.601384 2582 scope.go:117] "RemoveContainer" containerID="b6d9ed67ed4bf44bb46418bd953b5a327115b45cae610fd73bfca408cf6c40ce" Jul 16 18:07:10.609131 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.609112 2582 scope.go:117] "RemoveContainer" containerID="6af61e574e20b7428a0bd93f1c06b9b8265144c16e26dc31c1db2a498beaf116" Jul 16 18:07:10.626991 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.626965 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lspcn\" (UniqueName: \"kubernetes.io/projected/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-kube-api-access-lspcn\") pod \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\" (UID: \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\") " Jul 16 18:07:10.627069 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.627029 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-kube-rbac-proxy-sar-config\") pod \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\" (UID: \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\") " Jul 16 18:07:10.627115 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.627099 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-proxy-tls\") pod \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\" (UID: \"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd\") " Jul 16 18:07:10.627445 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.627412 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" (UID: "aad71ae1-a1c9-40ad-9fc8-5f0733be67bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:07:10.629279 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.629248 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-kube-api-access-lspcn" (OuterVolumeSpecName: "kube-api-access-lspcn") pod "aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" (UID: "aad71ae1-a1c9-40ad-9fc8-5f0733be67bd"). InnerVolumeSpecName "kube-api-access-lspcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:07:10.629378 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.629352 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" (UID: "aad71ae1-a1c9-40ad-9fc8-5f0733be67bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:07:10.727871 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.727840 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lspcn\" (UniqueName: \"kubernetes.io/projected/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-kube-api-access-lspcn\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:07:10.727871 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.727869 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:07:10.728077 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:10.727879 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:07:11.093741 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:11.093704 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" event={"ID":"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd","Type":"ContainerDied","Data":"6af61e574e20b7428a0bd93f1c06b9b8265144c16e26dc31c1db2a498beaf116"} Jul 16 18:07:11.094121 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:11.093769 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" event={"ID":"aad71ae1-a1c9-40ad-9fc8-5f0733be67bd","Type":"ContainerDied","Data":"858ae080a50577373c2509c4a351a2a6e3160b299009a74920774e22b3b33c7d"} Jul 16 18:07:11.094121 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:11.093717 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb" Jul 16 18:07:11.123282 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:11.123251 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb"] Jul 16 18:07:11.127401 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:11.127374 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a90bb-predictor-55f9ccb694-4vvcb"] Jul 16 18:07:12.588895 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:12.588846 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" path="/var/lib/kubelet/pods/aad71ae1-a1c9-40ad-9fc8-5f0733be67bd/volumes" Jul 16 18:07:14.087683 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:14.087653 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:14.088173 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:14.088028 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 16 18:07:14.088733 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:14.088716 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:14.089123 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:14.089097 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Jul 16 18:07:15.927552 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:15.927523 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:07:16.932617 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:16.932589 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:07:24.088158 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:24.088111 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 16 18:07:24.089706 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:24.089679 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Jul 16 18:07:34.088108 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:34.088063 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 16 18:07:34.089253 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:34.089223 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Jul 16 18:07:37.420868 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.420828 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld"] Jul 16 18:07:37.421291 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.421052 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kserve-container" containerID="cri-o://84de2af262c92efa9b26c1715da18eb73b3c85f2e23ebd4581aa993de896090a" gracePeriod=30 Jul 16 18:07:37.421291 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.421113 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kube-rbac-proxy" containerID="cri-o://63a4bcc0d748e56dab8b7dbb0f7fa0b399070b6b08f46abf3b2c15f1d5f3a269" gracePeriod=30 Jul 16 18:07:37.470629 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.470599 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm"] Jul 16 18:07:37.470900 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.470864 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kserve-container" containerID="cri-o://9f5ae837da361e8ff0d862efbc56f176e19cbb11f7993049eb3036544a2e07a6" gracePeriod=30 Jul 16 18:07:37.471008 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.470936 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kube-rbac-proxy" containerID="cri-o://d7c219f862b8fe962ceacb21ae3bb3428353d6d89b28520b56996430158b2035" gracePeriod=30 Jul 16 18:07:37.526689 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.526662 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n"] Jul 16 18:07:37.526995 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.526983 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kserve-container" Jul 16 18:07:37.527042 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.526997 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kserve-container" Jul 16 18:07:37.527042 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.527010 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kube-rbac-proxy" Jul 16 18:07:37.527042 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.527016 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kube-rbac-proxy" Jul 16 18:07:37.527042 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.527023 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kube-rbac-proxy" Jul 16 18:07:37.527042 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.527029 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kube-rbac-proxy" Jul 16 18:07:37.527042 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.527036 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kserve-container" Jul 16 18:07:37.527042 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.527041 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kserve-container" Jul 16 18:07:37.527600 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.527091 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kube-rbac-proxy" Jul 16 18:07:37.527600 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.527099 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad71ae1-a1c9-40ad-9fc8-5f0733be67bd" containerName="kserve-container" Jul 16 18:07:37.527600 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.527109 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kserve-container" Jul 16 18:07:37.527600 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.527115 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d38b5b-b79d-496b-bbd5-f0baa7173598" containerName="kube-rbac-proxy" Jul 16 18:07:37.531465 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.531442 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:37.533763 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.533739 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-21cc5-predictor-serving-cert\"" Jul 16 18:07:37.533878 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.533744 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-21cc5-kube-rbac-proxy-sar-config\"" Jul 16 18:07:37.538525 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.538464 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n"] Jul 16 18:07:37.621951 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.621919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2f85123-b57b-4560-b655-029db7a3ce32-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-21cc5-predictor-76676b5555-2rb8n\" (UID: \"c2f85123-b57b-4560-b655-029db7a3ce32\") " pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:37.622216 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.622127 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2f85123-b57b-4560-b655-029db7a3ce32-proxy-tls\") pod \"success-200-isvc-21cc5-predictor-76676b5555-2rb8n\" (UID: \"c2f85123-b57b-4560-b655-029db7a3ce32\") " pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:37.622321 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.622242 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glx9x\" (UniqueName: \"kubernetes.io/projected/c2f85123-b57b-4560-b655-029db7a3ce32-kube-api-access-glx9x\") pod \"success-200-isvc-21cc5-predictor-76676b5555-2rb8n\" (UID: \"c2f85123-b57b-4560-b655-029db7a3ce32\") " pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:37.631793 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.631768 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7"] Jul 16 18:07:37.635042 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.635026 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:37.637362 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.637269 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-21cc5-predictor-serving-cert\"" Jul 16 18:07:37.637498 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.637409 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-21cc5-kube-rbac-proxy-sar-config\"" Jul 16 18:07:37.646289 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.646269 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7"] Jul 16 18:07:37.723682 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.723609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2817fb1c-1f8c-4eda-86be-79a4078085ab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7\" (UID: \"2817fb1c-1f8c-4eda-86be-79a4078085ab\") " pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:37.723682 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.723645 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pdg9\" (UniqueName: \"kubernetes.io/projected/2817fb1c-1f8c-4eda-86be-79a4078085ab-kube-api-access-2pdg9\") pod \"error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7\" (UID: \"2817fb1c-1f8c-4eda-86be-79a4078085ab\") " pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:37.723854 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.723694 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2817fb1c-1f8c-4eda-86be-79a4078085ab-proxy-tls\") pod \"error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7\" (UID: \"2817fb1c-1f8c-4eda-86be-79a4078085ab\") " pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:37.723854 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.723716 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2f85123-b57b-4560-b655-029db7a3ce32-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-21cc5-predictor-76676b5555-2rb8n\" (UID: \"c2f85123-b57b-4560-b655-029db7a3ce32\") " pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:37.723854 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.723745 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2f85123-b57b-4560-b655-029db7a3ce32-proxy-tls\") pod \"success-200-isvc-21cc5-predictor-76676b5555-2rb8n\" (UID: \"c2f85123-b57b-4560-b655-029db7a3ce32\") " pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:37.723854 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.723805 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glx9x\" (UniqueName: \"kubernetes.io/projected/c2f85123-b57b-4560-b655-029db7a3ce32-kube-api-access-glx9x\") pod \"success-200-isvc-21cc5-predictor-76676b5555-2rb8n\" (UID: \"c2f85123-b57b-4560-b655-029db7a3ce32\") " pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:37.723854 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:07:37.723822 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-serving-cert: secret "success-200-isvc-21cc5-predictor-serving-cert" not found Jul 16 18:07:37.724055 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:07:37.723863 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f85123-b57b-4560-b655-029db7a3ce32-proxy-tls podName:c2f85123-b57b-4560-b655-029db7a3ce32 nodeName:}" failed. No retries permitted until 2026-07-16 18:07:38.223848946 +0000 UTC m=+2008.241078268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c2f85123-b57b-4560-b655-029db7a3ce32-proxy-tls") pod "success-200-isvc-21cc5-predictor-76676b5555-2rb8n" (UID: "c2f85123-b57b-4560-b655-029db7a3ce32") : secret "success-200-isvc-21cc5-predictor-serving-cert" not found Jul 16 18:07:37.724286 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.724270 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2f85123-b57b-4560-b655-029db7a3ce32-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-21cc5-predictor-76676b5555-2rb8n\" (UID: \"c2f85123-b57b-4560-b655-029db7a3ce32\") " pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:37.736703 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.736674 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glx9x\" (UniqueName: \"kubernetes.io/projected/c2f85123-b57b-4560-b655-029db7a3ce32-kube-api-access-glx9x\") pod \"success-200-isvc-21cc5-predictor-76676b5555-2rb8n\" (UID: \"c2f85123-b57b-4560-b655-029db7a3ce32\") " pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:37.824624 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.824595 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2817fb1c-1f8c-4eda-86be-79a4078085ab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7\" (UID: \"2817fb1c-1f8c-4eda-86be-79a4078085ab\") " pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:37.824763 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.824633 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdg9\" (UniqueName: \"kubernetes.io/projected/2817fb1c-1f8c-4eda-86be-79a4078085ab-kube-api-access-2pdg9\") pod \"error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7\" (UID: \"2817fb1c-1f8c-4eda-86be-79a4078085ab\") " pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:37.824763 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.824707 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2817fb1c-1f8c-4eda-86be-79a4078085ab-proxy-tls\") pod \"error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7\" (UID: \"2817fb1c-1f8c-4eda-86be-79a4078085ab\") " pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:37.825211 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.825186 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2817fb1c-1f8c-4eda-86be-79a4078085ab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7\" (UID: \"2817fb1c-1f8c-4eda-86be-79a4078085ab\") " pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:37.827120 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.827099 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2817fb1c-1f8c-4eda-86be-79a4078085ab-proxy-tls\") pod \"error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7\" (UID: \"2817fb1c-1f8c-4eda-86be-79a4078085ab\") " pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:37.835574 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.835553 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pdg9\" (UniqueName: \"kubernetes.io/projected/2817fb1c-1f8c-4eda-86be-79a4078085ab-kube-api-access-2pdg9\") pod \"error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7\" (UID: \"2817fb1c-1f8c-4eda-86be-79a4078085ab\") " pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:37.945641 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:37.945614 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:38.075941 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.075919 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7"] Jul 16 18:07:38.212921 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.212888 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" event={"ID":"2817fb1c-1f8c-4eda-86be-79a4078085ab","Type":"ContainerStarted","Data":"83649090fe36c7ef9e728aef5fd87aa3eff52a66bce69589fb85928f00394b9e"} Jul 16 18:07:38.213071 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.212971 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:38.213071 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.213005 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" event={"ID":"2817fb1c-1f8c-4eda-86be-79a4078085ab","Type":"ContainerStarted","Data":"98ac93a70f7853fcb3f6984efd59f3b5b392a3cf6a88b6c3dc702b1766f1d634"} Jul 16 18:07:38.213071 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.213034 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" event={"ID":"2817fb1c-1f8c-4eda-86be-79a4078085ab","Type":"ContainerStarted","Data":"dc1d002fb8ccafbd8d2da60ca1eb2c3ccdc8e3f78c1da8c5355dcd60e5443a9d"} Jul 16 18:07:38.213071 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.213061 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:38.214295 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.214265 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" Jul 16 18:07:38.214687 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.214664 2582 generic.go:356] "Generic (PLEG): container finished" podID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerID="63a4bcc0d748e56dab8b7dbb0f7fa0b399070b6b08f46abf3b2c15f1d5f3a269" exitCode=2 Jul 16 18:07:38.214777 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.214727 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" event={"ID":"d8f53a82-a7e5-4703-9241-d93ff28b09c6","Type":"ContainerDied","Data":"63a4bcc0d748e56dab8b7dbb0f7fa0b399070b6b08f46abf3b2c15f1d5f3a269"} Jul 16 18:07:38.216312 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.216293 2582 generic.go:356] "Generic (PLEG): container finished" podID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerID="d7c219f862b8fe962ceacb21ae3bb3428353d6d89b28520b56996430158b2035" exitCode=2 Jul 16 18:07:38.216403 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.216335 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" event={"ID":"02e68e74-6ad3-43ac-bd1d-dabe89c07510","Type":"ContainerDied","Data":"d7c219f862b8fe962ceacb21ae3bb3428353d6d89b28520b56996430158b2035"} Jul 16 18:07:38.228558 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.228540 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2f85123-b57b-4560-b655-029db7a3ce32-proxy-tls\") pod \"success-200-isvc-21cc5-predictor-76676b5555-2rb8n\" (UID: \"c2f85123-b57b-4560-b655-029db7a3ce32\") " pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:38.230785 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.230762 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2f85123-b57b-4560-b655-029db7a3ce32-proxy-tls\") pod \"success-200-isvc-21cc5-predictor-76676b5555-2rb8n\" (UID: \"c2f85123-b57b-4560-b655-029db7a3ce32\") " pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:38.248019 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.247948 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" podStartSLOduration=1.247936004 podStartE2EDuration="1.247936004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:07:38.247693361 +0000 UTC m=+2008.264922703" watchObservedRunningTime="2026-07-16 18:07:38.247936004 +0000 UTC m=+2008.265165345" Jul 16 18:07:38.443070 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.443041 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:38.575124 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:38.575089 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n"] Jul 16 18:07:38.576713 ip-10-0-139-103 kubenswrapper[2582]: W0716 18:07:38.576687 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f85123_b57b_4560_b655_029db7a3ce32.slice/crio-11012d0dfcaad0cad59ae673eac1f7145f8bd5f06b81ffa0f057f3014f0b2578 WatchSource:0}: Error finding container 11012d0dfcaad0cad59ae673eac1f7145f8bd5f06b81ffa0f057f3014f0b2578: Status 404 returned error can't find the container with id 11012d0dfcaad0cad59ae673eac1f7145f8bd5f06b81ffa0f057f3014f0b2578 Jul 16 18:07:39.221878 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:39.221835 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" event={"ID":"c2f85123-b57b-4560-b655-029db7a3ce32","Type":"ContainerStarted","Data":"a7961e976cf77b9adb67769df3a8b30c80f3161630ea77c6c2395aa5ebd7b4e4"} Jul 16 18:07:39.222066 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:39.222022 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:39.222124 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:39.222074 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" event={"ID":"c2f85123-b57b-4560-b655-029db7a3ce32","Type":"ContainerStarted","Data":"5af61c209aef34479bc145a37617abe22017a0170406e20b3c64dd4843436195"} Jul 16 18:07:39.222186 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:39.222111 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" event={"ID":"c2f85123-b57b-4560-b655-029db7a3ce32","Type":"ContainerStarted","Data":"11012d0dfcaad0cad59ae673eac1f7145f8bd5f06b81ffa0f057f3014f0b2578"} Jul 16 18:07:39.222282 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:39.222255 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" Jul 16 18:07:39.222350 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:39.222309 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:39.223411 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:39.223388 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 16 18:07:39.240379 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:39.240327 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" podStartSLOduration=2.240315768 podStartE2EDuration="2.240315768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:07:39.23886549 +0000 UTC m=+2009.256094855" watchObservedRunningTime="2026-07-16 18:07:39.240315768 +0000 UTC m=+2009.257545109" Jul 16 18:07:40.226407 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:40.226369 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 16 18:07:40.921853 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:40.921807 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused" Jul 16 18:07:41.233228 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.233187 2582 generic.go:356] "Generic (PLEG): container finished" podID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerID="84de2af262c92efa9b26c1715da18eb73b3c85f2e23ebd4581aa993de896090a" exitCode=0 Jul 16 18:07:41.233719 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.233265 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" event={"ID":"d8f53a82-a7e5-4703-9241-d93ff28b09c6","Type":"ContainerDied","Data":"84de2af262c92efa9b26c1715da18eb73b3c85f2e23ebd4581aa993de896090a"} Jul 16 18:07:41.235349 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.235321 2582 generic.go:356] "Generic (PLEG): container finished" podID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerID="9f5ae837da361e8ff0d862efbc56f176e19cbb11f7993049eb3036544a2e07a6" exitCode=0 Jul 16 18:07:41.235631 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.235336 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" event={"ID":"02e68e74-6ad3-43ac-bd1d-dabe89c07510","Type":"ContainerDied","Data":"9f5ae837da361e8ff0d862efbc56f176e19cbb11f7993049eb3036544a2e07a6"} Jul 16 18:07:41.236081 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.236062 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 16 18:07:41.340673 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.340652 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:07:41.343342 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.343325 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:07:41.454287 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.454203 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f53a82-a7e5-4703-9241-d93ff28b09c6-proxy-tls\") pod \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\" (UID: \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\") " Jul 16 18:07:41.454287 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.454253 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e68e74-6ad3-43ac-bd1d-dabe89c07510-proxy-tls\") pod \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " Jul 16 18:07:41.454287 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.454283 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tktz\" (UniqueName: \"kubernetes.io/projected/02e68e74-6ad3-43ac-bd1d-dabe89c07510-kube-api-access-5tktz\") pod \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " Jul 16 18:07:41.454602 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.454307 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f53a82-a7e5-4703-9241-d93ff28b09c6-kube-rbac-proxy-sar-config\") pod \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\" (UID: \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\") " Jul 16 18:07:41.454602 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.454366 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e68e74-6ad3-43ac-bd1d-dabe89c07510-kube-rbac-proxy-sar-config\") pod \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\" (UID: \"02e68e74-6ad3-43ac-bd1d-dabe89c07510\") " Jul 16 18:07:41.454602 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.454394 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc57g\" (UniqueName: \"kubernetes.io/projected/d8f53a82-a7e5-4703-9241-d93ff28b09c6-kube-api-access-mc57g\") pod \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\" (UID: \"d8f53a82-a7e5-4703-9241-d93ff28b09c6\") " Jul 16 18:07:41.454759 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.454737 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f53a82-a7e5-4703-9241-d93ff28b09c6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d8f53a82-a7e5-4703-9241-d93ff28b09c6" (UID: "d8f53a82-a7e5-4703-9241-d93ff28b09c6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:07:41.454799 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.454773 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e68e74-6ad3-43ac-bd1d-dabe89c07510-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "02e68e74-6ad3-43ac-bd1d-dabe89c07510" (UID: "02e68e74-6ad3-43ac-bd1d-dabe89c07510"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:07:41.456595 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.456564 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e68e74-6ad3-43ac-bd1d-dabe89c07510-kube-api-access-5tktz" (OuterVolumeSpecName: "kube-api-access-5tktz") pod "02e68e74-6ad3-43ac-bd1d-dabe89c07510" (UID: "02e68e74-6ad3-43ac-bd1d-dabe89c07510"). InnerVolumeSpecName "kube-api-access-5tktz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:07:41.456736 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.456653 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f53a82-a7e5-4703-9241-d93ff28b09c6-kube-api-access-mc57g" (OuterVolumeSpecName: "kube-api-access-mc57g") pod "d8f53a82-a7e5-4703-9241-d93ff28b09c6" (UID: "d8f53a82-a7e5-4703-9241-d93ff28b09c6"). InnerVolumeSpecName "kube-api-access-mc57g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:07:41.457032 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.457013 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f53a82-a7e5-4703-9241-d93ff28b09c6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8f53a82-a7e5-4703-9241-d93ff28b09c6" (UID: "d8f53a82-a7e5-4703-9241-d93ff28b09c6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:07:41.457090 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.457068 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e68e74-6ad3-43ac-bd1d-dabe89c07510-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02e68e74-6ad3-43ac-bd1d-dabe89c07510" (UID: "02e68e74-6ad3-43ac-bd1d-dabe89c07510"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:07:41.555751 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.555715 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e68e74-6ad3-43ac-bd1d-dabe89c07510-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:07:41.555751 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.555744 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc57g\" (UniqueName: \"kubernetes.io/projected/d8f53a82-a7e5-4703-9241-d93ff28b09c6-kube-api-access-mc57g\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:07:41.555751 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.555755 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f53a82-a7e5-4703-9241-d93ff28b09c6-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:07:41.555751 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.555764 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e68e74-6ad3-43ac-bd1d-dabe89c07510-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:07:41.556014 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.555772 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tktz\" (UniqueName: \"kubernetes.io/projected/02e68e74-6ad3-43ac-bd1d-dabe89c07510-kube-api-access-5tktz\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:07:41.556014 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:41.555780 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f53a82-a7e5-4703-9241-d93ff28b09c6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:07:42.240678 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.240647 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" Jul 16 18:07:42.241127 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.240647 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld" event={"ID":"d8f53a82-a7e5-4703-9241-d93ff28b09c6","Type":"ContainerDied","Data":"eea009037f3a3e84e88d4c0261deed32169bf3cc1dd3ccea06356a1f025c2f34"} Jul 16 18:07:42.241127 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.240788 2582 scope.go:117] "RemoveContainer" containerID="63a4bcc0d748e56dab8b7dbb0f7fa0b399070b6b08f46abf3b2c15f1d5f3a269" Jul 16 18:07:42.249121 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.249060 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" event={"ID":"02e68e74-6ad3-43ac-bd1d-dabe89c07510","Type":"ContainerDied","Data":"e24653c7c9e96594af34c9cee0ba1e1cc8774d4cc9e579f76af180df05c52c4a"} Jul 16 18:07:42.249278 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.249148 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm" Jul 16 18:07:42.255838 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.255822 2582 scope.go:117] "RemoveContainer" containerID="84de2af262c92efa9b26c1715da18eb73b3c85f2e23ebd4581aa993de896090a" Jul 16 18:07:42.263427 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.263407 2582 scope.go:117] "RemoveContainer" containerID="d7c219f862b8fe962ceacb21ae3bb3428353d6d89b28520b56996430158b2035" Jul 16 18:07:42.266447 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.266427 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld"] Jul 16 18:07:42.270643 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.270621 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a019b-predictor-5578757556-bfcld"] Jul 16 18:07:42.270946 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.270935 2582 scope.go:117] "RemoveContainer" containerID="9f5ae837da361e8ff0d862efbc56f176e19cbb11f7993049eb3036544a2e07a6" Jul 16 18:07:42.282750 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.282727 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm"] Jul 16 18:07:42.286415 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.286390 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a019b-predictor-54d5fc55df-ccngm"] Jul 16 18:07:42.587755 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.587681 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" path="/var/lib/kubelet/pods/02e68e74-6ad3-43ac-bd1d-dabe89c07510/volumes" Jul 16 18:07:42.588093 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:42.588080 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" path="/var/lib/kubelet/pods/d8f53a82-a7e5-4703-9241-d93ff28b09c6/volumes" Jul 16 18:07:44.088144 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:44.088105 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 16 18:07:44.089257 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:44.089219 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Jul 16 18:07:44.226919 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:44.226887 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:07:44.227542 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:44.227509 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" Jul 16 18:07:46.240840 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:46.240811 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:07:46.241236 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:46.241211 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 16 18:07:54.088617 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:54.088586 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:07:54.089631 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:54.089612 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:07:54.227610 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:54.227561 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" Jul 16 18:07:56.241264 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:07:56.241219 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 16 18:08:04.227435 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:08:04.227398 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" Jul 16 18:08:06.241852 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:08:06.241812 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 16 18:08:14.227465 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:08:14.227420 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" Jul 16 18:08:16.241389 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:08:16.241349 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 16 18:08:24.228531 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:08:24.228494 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:08:26.242207 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:08:26.242177 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:09:10.533643 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:09:10.533610 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 18:09:10.539811 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:09:10.539786 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 18:09:10.540050 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:09:10.540031 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 18:09:10.545704 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:09:10.545686 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 18:14:10.556143 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:14:10.556108 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 18:14:10.562217 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:14:10.562179 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 18:14:10.563921 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:14:10.563903 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 18:14:10.569217 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:14:10.569203 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 18:16:52.389890 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:52.389857 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n"] Jul 16 18:16:52.390400 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:52.390092 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kserve-container" containerID="cri-o://5af61c209aef34479bc145a37617abe22017a0170406e20b3c64dd4843436195" gracePeriod=30 Jul 16 18:16:52.390400 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:52.390141 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kube-rbac-proxy" containerID="cri-o://a7961e976cf77b9adb67769df3a8b30c80f3161630ea77c6c2395aa5ebd7b4e4" gracePeriod=30 Jul 16 18:16:52.428574 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:52.428549 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7"] Jul 16 18:16:52.428802 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:52.428761 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kserve-container" containerID="cri-o://98ac93a70f7853fcb3f6984efd59f3b5b392a3cf6a88b6c3dc702b1766f1d634" gracePeriod=30 Jul 16 18:16:52.428921 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:52.428792 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kube-rbac-proxy" containerID="cri-o://83649090fe36c7ef9e728aef5fd87aa3eff52a66bce69589fb85928f00394b9e" gracePeriod=30 Jul 16 18:16:53.354527 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:53.354489 2582 generic.go:356] "Generic (PLEG): container finished" podID="c2f85123-b57b-4560-b655-029db7a3ce32" containerID="a7961e976cf77b9adb67769df3a8b30c80f3161630ea77c6c2395aa5ebd7b4e4" exitCode=2 Jul 16 18:16:53.354527 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:53.354509 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" event={"ID":"c2f85123-b57b-4560-b655-029db7a3ce32","Type":"ContainerDied","Data":"a7961e976cf77b9adb67769df3a8b30c80f3161630ea77c6c2395aa5ebd7b4e4"} Jul 16 18:16:53.356245 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:53.356222 2582 generic.go:356] "Generic (PLEG): container finished" podID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerID="83649090fe36c7ef9e728aef5fd87aa3eff52a66bce69589fb85928f00394b9e" exitCode=2 Jul 16 18:16:53.356361 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:53.356293 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" event={"ID":"2817fb1c-1f8c-4eda-86be-79a4078085ab","Type":"ContainerDied","Data":"83649090fe36c7ef9e728aef5fd87aa3eff52a66bce69589fb85928f00394b9e"} Jul 16 18:16:54.222640 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:54.222600 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused" Jul 16 18:16:54.227767 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:54.227743 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" Jul 16 18:16:55.285657 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.285637 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:16:55.288548 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.288530 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:16:55.365021 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.364964 2582 generic.go:356] "Generic (PLEG): container finished" podID="c2f85123-b57b-4560-b655-029db7a3ce32" containerID="5af61c209aef34479bc145a37617abe22017a0170406e20b3c64dd4843436195" exitCode=0 Jul 16 18:16:55.365021 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.365007 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" Jul 16 18:16:55.365184 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.365047 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" event={"ID":"c2f85123-b57b-4560-b655-029db7a3ce32","Type":"ContainerDied","Data":"5af61c209aef34479bc145a37617abe22017a0170406e20b3c64dd4843436195"} Jul 16 18:16:55.365184 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.365107 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n" event={"ID":"c2f85123-b57b-4560-b655-029db7a3ce32","Type":"ContainerDied","Data":"11012d0dfcaad0cad59ae673eac1f7145f8bd5f06b81ffa0f057f3014f0b2578"} Jul 16 18:16:55.365184 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.365135 2582 scope.go:117] "RemoveContainer" containerID="a7961e976cf77b9adb67769df3a8b30c80f3161630ea77c6c2395aa5ebd7b4e4" Jul 16 18:16:55.366687 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.366625 2582 generic.go:356] "Generic (PLEG): container finished" podID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerID="98ac93a70f7853fcb3f6984efd59f3b5b392a3cf6a88b6c3dc702b1766f1d634" exitCode=0 Jul 16 18:16:55.366687 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.366658 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" event={"ID":"2817fb1c-1f8c-4eda-86be-79a4078085ab","Type":"ContainerDied","Data":"98ac93a70f7853fcb3f6984efd59f3b5b392a3cf6a88b6c3dc702b1766f1d634"} Jul 16 18:16:55.366687 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.366669 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" Jul 16 18:16:55.366903 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.366702 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7" event={"ID":"2817fb1c-1f8c-4eda-86be-79a4078085ab","Type":"ContainerDied","Data":"dc1d002fb8ccafbd8d2da60ca1eb2c3ccdc8e3f78c1da8c5355dcd60e5443a9d"} Jul 16 18:16:55.373321 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.373305 2582 scope.go:117] "RemoveContainer" containerID="5af61c209aef34479bc145a37617abe22017a0170406e20b3c64dd4843436195" Jul 16 18:16:55.380356 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.380339 2582 scope.go:117] "RemoveContainer" containerID="a7961e976cf77b9adb67769df3a8b30c80f3161630ea77c6c2395aa5ebd7b4e4" Jul 16 18:16:55.380626 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:16:55.380605 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7961e976cf77b9adb67769df3a8b30c80f3161630ea77c6c2395aa5ebd7b4e4\": container with ID starting with a7961e976cf77b9adb67769df3a8b30c80f3161630ea77c6c2395aa5ebd7b4e4 not found: ID does not exist" containerID="a7961e976cf77b9adb67769df3a8b30c80f3161630ea77c6c2395aa5ebd7b4e4" Jul 16 18:16:55.380675 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.380634 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7961e976cf77b9adb67769df3a8b30c80f3161630ea77c6c2395aa5ebd7b4e4"} err="failed to get container status \"a7961e976cf77b9adb67769df3a8b30c80f3161630ea77c6c2395aa5ebd7b4e4\": rpc error: code = NotFound desc = could not find container \"a7961e976cf77b9adb67769df3a8b30c80f3161630ea77c6c2395aa5ebd7b4e4\": container with ID starting with a7961e976cf77b9adb67769df3a8b30c80f3161630ea77c6c2395aa5ebd7b4e4 not found: ID does not exist" Jul 16 18:16:55.380675 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.380650 2582 scope.go:117] "RemoveContainer" containerID="5af61c209aef34479bc145a37617abe22017a0170406e20b3c64dd4843436195" Jul 16 18:16:55.380892 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:16:55.380876 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af61c209aef34479bc145a37617abe22017a0170406e20b3c64dd4843436195\": container with ID starting with 5af61c209aef34479bc145a37617abe22017a0170406e20b3c64dd4843436195 not found: ID does not exist" containerID="5af61c209aef34479bc145a37617abe22017a0170406e20b3c64dd4843436195" Jul 16 18:16:55.380943 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.380896 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af61c209aef34479bc145a37617abe22017a0170406e20b3c64dd4843436195"} err="failed to get container status \"5af61c209aef34479bc145a37617abe22017a0170406e20b3c64dd4843436195\": rpc error: code = NotFound desc = could not find container \"5af61c209aef34479bc145a37617abe22017a0170406e20b3c64dd4843436195\": container with ID starting with 5af61c209aef34479bc145a37617abe22017a0170406e20b3c64dd4843436195 not found: ID does not exist" Jul 16 18:16:55.380943 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.380910 2582 scope.go:117] "RemoveContainer" containerID="83649090fe36c7ef9e728aef5fd87aa3eff52a66bce69589fb85928f00394b9e" Jul 16 18:16:55.385932 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.385913 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2f85123-b57b-4560-b655-029db7a3ce32-kube-rbac-proxy-sar-config\") pod \"c2f85123-b57b-4560-b655-029db7a3ce32\" (UID: \"c2f85123-b57b-4560-b655-029db7a3ce32\") " Jul 16 18:16:55.386020 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.385941 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2817fb1c-1f8c-4eda-86be-79a4078085ab-kube-rbac-proxy-sar-config\") pod \"2817fb1c-1f8c-4eda-86be-79a4078085ab\" (UID: \"2817fb1c-1f8c-4eda-86be-79a4078085ab\") " Jul 16 18:16:55.386020 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.385991 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glx9x\" (UniqueName: \"kubernetes.io/projected/c2f85123-b57b-4560-b655-029db7a3ce32-kube-api-access-glx9x\") pod \"c2f85123-b57b-4560-b655-029db7a3ce32\" (UID: \"c2f85123-b57b-4560-b655-029db7a3ce32\") " Jul 16 18:16:55.386111 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.386026 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2817fb1c-1f8c-4eda-86be-79a4078085ab-proxy-tls\") pod \"2817fb1c-1f8c-4eda-86be-79a4078085ab\" (UID: \"2817fb1c-1f8c-4eda-86be-79a4078085ab\") " Jul 16 18:16:55.386111 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.386060 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2f85123-b57b-4560-b655-029db7a3ce32-proxy-tls\") pod \"c2f85123-b57b-4560-b655-029db7a3ce32\" (UID: \"c2f85123-b57b-4560-b655-029db7a3ce32\") " Jul 16 18:16:55.386221 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.386137 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pdg9\" (UniqueName: \"kubernetes.io/projected/2817fb1c-1f8c-4eda-86be-79a4078085ab-kube-api-access-2pdg9\") pod \"2817fb1c-1f8c-4eda-86be-79a4078085ab\" (UID: \"2817fb1c-1f8c-4eda-86be-79a4078085ab\") " Jul 16 18:16:55.386309 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.386285 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f85123-b57b-4560-b655-029db7a3ce32-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c2f85123-b57b-4560-b655-029db7a3ce32" (UID: "c2f85123-b57b-4560-b655-029db7a3ce32"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:16:55.386370 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.386334 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2817fb1c-1f8c-4eda-86be-79a4078085ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2817fb1c-1f8c-4eda-86be-79a4078085ab" (UID: "2817fb1c-1f8c-4eda-86be-79a4078085ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:16:55.386438 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.386389 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2f85123-b57b-4560-b655-029db7a3ce32-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:16:55.386438 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.386401 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2817fb1c-1f8c-4eda-86be-79a4078085ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:16:55.388252 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.388231 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f85123-b57b-4560-b655-029db7a3ce32-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2f85123-b57b-4560-b655-029db7a3ce32" (UID: "c2f85123-b57b-4560-b655-029db7a3ce32"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:16:55.388354 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.388332 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2817fb1c-1f8c-4eda-86be-79a4078085ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2817fb1c-1f8c-4eda-86be-79a4078085ab" (UID: "2817fb1c-1f8c-4eda-86be-79a4078085ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:16:55.388585 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.388541 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2817fb1c-1f8c-4eda-86be-79a4078085ab-kube-api-access-2pdg9" (OuterVolumeSpecName: "kube-api-access-2pdg9") pod "2817fb1c-1f8c-4eda-86be-79a4078085ab" (UID: "2817fb1c-1f8c-4eda-86be-79a4078085ab"). InnerVolumeSpecName "kube-api-access-2pdg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:16:55.388629 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.388598 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f85123-b57b-4560-b655-029db7a3ce32-kube-api-access-glx9x" (OuterVolumeSpecName: "kube-api-access-glx9x") pod "c2f85123-b57b-4560-b655-029db7a3ce32" (UID: "c2f85123-b57b-4560-b655-029db7a3ce32"). InnerVolumeSpecName "kube-api-access-glx9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:16:55.388674 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.388626 2582 scope.go:117] "RemoveContainer" containerID="98ac93a70f7853fcb3f6984efd59f3b5b392a3cf6a88b6c3dc702b1766f1d634" Jul 16 18:16:55.398050 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.398017 2582 scope.go:117] "RemoveContainer" containerID="83649090fe36c7ef9e728aef5fd87aa3eff52a66bce69589fb85928f00394b9e" Jul 16 18:16:55.398272 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:16:55.398254 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83649090fe36c7ef9e728aef5fd87aa3eff52a66bce69589fb85928f00394b9e\": container with ID starting with 83649090fe36c7ef9e728aef5fd87aa3eff52a66bce69589fb85928f00394b9e not found: ID does not exist" containerID="83649090fe36c7ef9e728aef5fd87aa3eff52a66bce69589fb85928f00394b9e" Jul 16 18:16:55.398329 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.398280 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83649090fe36c7ef9e728aef5fd87aa3eff52a66bce69589fb85928f00394b9e"} err="failed to get container status \"83649090fe36c7ef9e728aef5fd87aa3eff52a66bce69589fb85928f00394b9e\": rpc error: code = NotFound desc = could not find container \"83649090fe36c7ef9e728aef5fd87aa3eff52a66bce69589fb85928f00394b9e\": container with ID starting with 83649090fe36c7ef9e728aef5fd87aa3eff52a66bce69589fb85928f00394b9e not found: ID does not exist" Jul 16 18:16:55.398329 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.398300 2582 scope.go:117] "RemoveContainer" containerID="98ac93a70f7853fcb3f6984efd59f3b5b392a3cf6a88b6c3dc702b1766f1d634" Jul 16 18:16:55.398648 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:16:55.398615 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ac93a70f7853fcb3f6984efd59f3b5b392a3cf6a88b6c3dc702b1766f1d634\": container with ID starting with 98ac93a70f7853fcb3f6984efd59f3b5b392a3cf6a88b6c3dc702b1766f1d634 not found: ID does not exist" containerID="98ac93a70f7853fcb3f6984efd59f3b5b392a3cf6a88b6c3dc702b1766f1d634" Jul 16 18:16:55.398698 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.398653 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ac93a70f7853fcb3f6984efd59f3b5b392a3cf6a88b6c3dc702b1766f1d634"} err="failed to get container status \"98ac93a70f7853fcb3f6984efd59f3b5b392a3cf6a88b6c3dc702b1766f1d634\": rpc error: code = NotFound desc = could not find container \"98ac93a70f7853fcb3f6984efd59f3b5b392a3cf6a88b6c3dc702b1766f1d634\": container with ID starting with 98ac93a70f7853fcb3f6984efd59f3b5b392a3cf6a88b6c3dc702b1766f1d634 not found: ID does not exist" Jul 16 18:16:55.487009 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.486985 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glx9x\" (UniqueName: \"kubernetes.io/projected/c2f85123-b57b-4560-b655-029db7a3ce32-kube-api-access-glx9x\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:16:55.487009 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.487005 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2817fb1c-1f8c-4eda-86be-79a4078085ab-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:16:55.487125 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.487015 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2f85123-b57b-4560-b655-029db7a3ce32-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:16:55.487125 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.487025 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pdg9\" (UniqueName: \"kubernetes.io/projected/2817fb1c-1f8c-4eda-86be-79a4078085ab-kube-api-access-2pdg9\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:16:55.689261 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.689235 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n"] Jul 16 18:16:55.691441 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.691423 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21cc5-predictor-76676b5555-2rb8n"] Jul 16 18:16:55.702359 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.702328 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7"] Jul 16 18:16:55.706258 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:55.706239 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21cc5-predictor-5d5c5f9fdb-xnxd7"] Jul 16 18:16:56.584580 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:56.584546 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" path="/var/lib/kubelet/pods/2817fb1c-1f8c-4eda-86be-79a4078085ab/volumes" Jul 16 18:16:56.584967 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:16:56.584953 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" path="/var/lib/kubelet/pods/c2f85123-b57b-4560-b655-029db7a3ce32/volumes" Jul 16 18:19:10.577766 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:19:10.577690 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 18:19:10.584504 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:19:10.584461 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 18:19:10.587463 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:19:10.587442 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 18:19:10.592586 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:19:10.592570 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 18:24:10.601310 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:10.601225 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 18:24:10.607137 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:10.607111 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 18:24:10.609427 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:10.609409 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 18:24:10.614387 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:10.614370 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 18:24:26.077983 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:26.077948 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw"] Jul 16 18:24:26.078389 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:26.078211 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kserve-container" containerID="cri-o://e35960aca8bde3ceace01f9d4c0a5e0a28ec79f3f091bd75c93595580d2e8bf5" gracePeriod=30 Jul 16 18:24:26.078389 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:26.078265 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kube-rbac-proxy" containerID="cri-o://173939769c60feb06844afb0d3776af5acb10ec442086fbd822eb98b792bbe51" gracePeriod=30 Jul 16 18:24:26.129461 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:26.129419 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g"] Jul 16 18:24:26.129687 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:26.129656 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kserve-container" containerID="cri-o://df4c5ea879819ab52f3fb68cf925d5c6695ee72bab4365b89becb372dc044ef6" gracePeriod=30 Jul 16 18:24:26.129825 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:26.129714 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kube-rbac-proxy" containerID="cri-o://092f13f82034a886a7356d5779b4f47ac0fbaaf6c000669b5abf1ed5e7bfb85a" gracePeriod=30 Jul 16 18:24:27.030357 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:27.030314 2582 generic.go:356] "Generic (PLEG): container finished" podID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerID="173939769c60feb06844afb0d3776af5acb10ec442086fbd822eb98b792bbe51" exitCode=2 Jul 16 18:24:27.030565 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:27.030379 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" event={"ID":"ef32f086-22fa-4544-a6e3-b6e27db85bfc","Type":"ContainerDied","Data":"173939769c60feb06844afb0d3776af5acb10ec442086fbd822eb98b792bbe51"} Jul 16 18:24:27.032034 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:27.032011 2582 generic.go:356] "Generic (PLEG): container finished" podID="ae46f104-584c-44ce-ae08-507d364f8971" containerID="092f13f82034a886a7356d5779b4f47ac0fbaaf6c000669b5abf1ed5e7bfb85a" exitCode=2 Jul 16 18:24:27.032122 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:27.032051 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" event={"ID":"ae46f104-584c-44ce-ae08-507d364f8971","Type":"ContainerDied","Data":"092f13f82034a886a7356d5779b4f47ac0fbaaf6c000669b5abf1ed5e7bfb85a"} Jul 16 18:24:28.937454 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:28.937399 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:24:28.971071 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:28.971049 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:24:29.008520 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.008494 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef32f086-22fa-4544-a6e3-b6e27db85bfc-proxy-tls\") pod \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\" (UID: \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\") " Jul 16 18:24:29.008670 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.008546 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef32f086-22fa-4544-a6e3-b6e27db85bfc-kube-rbac-proxy-sar-config\") pod \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\" (UID: \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\") " Jul 16 18:24:29.008670 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.008602 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cb9m\" (UniqueName: \"kubernetes.io/projected/ef32f086-22fa-4544-a6e3-b6e27db85bfc-kube-api-access-6cb9m\") pod \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\" (UID: \"ef32f086-22fa-4544-a6e3-b6e27db85bfc\") " Jul 16 18:24:29.008945 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.008919 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef32f086-22fa-4544-a6e3-b6e27db85bfc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef32f086-22fa-4544-a6e3-b6e27db85bfc" (UID: "ef32f086-22fa-4544-a6e3-b6e27db85bfc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:24:29.010829 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.010803 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef32f086-22fa-4544-a6e3-b6e27db85bfc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef32f086-22fa-4544-a6e3-b6e27db85bfc" (UID: "ef32f086-22fa-4544-a6e3-b6e27db85bfc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:24:29.010915 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.010851 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef32f086-22fa-4544-a6e3-b6e27db85bfc-kube-api-access-6cb9m" (OuterVolumeSpecName: "kube-api-access-6cb9m") pod "ef32f086-22fa-4544-a6e3-b6e27db85bfc" (UID: "ef32f086-22fa-4544-a6e3-b6e27db85bfc"). InnerVolumeSpecName "kube-api-access-6cb9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:24:29.039660 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.039634 2582 generic.go:356] "Generic (PLEG): container finished" podID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerID="e35960aca8bde3ceace01f9d4c0a5e0a28ec79f3f091bd75c93595580d2e8bf5" exitCode=0 Jul 16 18:24:29.039780 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.039665 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" event={"ID":"ef32f086-22fa-4544-a6e3-b6e27db85bfc","Type":"ContainerDied","Data":"e35960aca8bde3ceace01f9d4c0a5e0a28ec79f3f091bd75c93595580d2e8bf5"} Jul 16 18:24:29.039780 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.039681 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" Jul 16 18:24:29.039780 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.039740 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw" event={"ID":"ef32f086-22fa-4544-a6e3-b6e27db85bfc","Type":"ContainerDied","Data":"78decdcdfa5242866aca1a624e611cb6d4961ac1bb99b7607abf6cd1d4a698dc"} Jul 16 18:24:29.039927 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.039783 2582 scope.go:117] "RemoveContainer" containerID="173939769c60feb06844afb0d3776af5acb10ec442086fbd822eb98b792bbe51" Jul 16 18:24:29.041352 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.041329 2582 generic.go:356] "Generic (PLEG): container finished" podID="ae46f104-584c-44ce-ae08-507d364f8971" containerID="df4c5ea879819ab52f3fb68cf925d5c6695ee72bab4365b89becb372dc044ef6" exitCode=0 Jul 16 18:24:29.041443 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.041360 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" event={"ID":"ae46f104-584c-44ce-ae08-507d364f8971","Type":"ContainerDied","Data":"df4c5ea879819ab52f3fb68cf925d5c6695ee72bab4365b89becb372dc044ef6"} Jul 16 18:24:29.041443 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.041378 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" Jul 16 18:24:29.041443 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.041424 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g" event={"ID":"ae46f104-584c-44ce-ae08-507d364f8971","Type":"ContainerDied","Data":"1846d5fa7f0a067a7d61397791c6c4d013251d63c960bd3dd29110ee08255049"} Jul 16 18:24:29.047889 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.047873 2582 scope.go:117] "RemoveContainer" containerID="e35960aca8bde3ceace01f9d4c0a5e0a28ec79f3f091bd75c93595580d2e8bf5" Jul 16 18:24:29.054727 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.054706 2582 scope.go:117] "RemoveContainer" containerID="173939769c60feb06844afb0d3776af5acb10ec442086fbd822eb98b792bbe51" Jul 16 18:24:29.054968 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:24:29.054951 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173939769c60feb06844afb0d3776af5acb10ec442086fbd822eb98b792bbe51\": container with ID starting with 173939769c60feb06844afb0d3776af5acb10ec442086fbd822eb98b792bbe51 not found: ID does not exist" containerID="173939769c60feb06844afb0d3776af5acb10ec442086fbd822eb98b792bbe51" Jul 16 18:24:29.055022 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.054975 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173939769c60feb06844afb0d3776af5acb10ec442086fbd822eb98b792bbe51"} err="failed to get container status \"173939769c60feb06844afb0d3776af5acb10ec442086fbd822eb98b792bbe51\": rpc error: code = NotFound desc = could not find container \"173939769c60feb06844afb0d3776af5acb10ec442086fbd822eb98b792bbe51\": container with ID starting with 173939769c60feb06844afb0d3776af5acb10ec442086fbd822eb98b792bbe51 not found: ID does not exist" Jul 16 18:24:29.055022 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.054989 2582 scope.go:117] "RemoveContainer" containerID="e35960aca8bde3ceace01f9d4c0a5e0a28ec79f3f091bd75c93595580d2e8bf5" Jul 16 18:24:29.055226 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:24:29.055208 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35960aca8bde3ceace01f9d4c0a5e0a28ec79f3f091bd75c93595580d2e8bf5\": container with ID starting with e35960aca8bde3ceace01f9d4c0a5e0a28ec79f3f091bd75c93595580d2e8bf5 not found: ID does not exist" containerID="e35960aca8bde3ceace01f9d4c0a5e0a28ec79f3f091bd75c93595580d2e8bf5" Jul 16 18:24:29.055276 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.055232 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35960aca8bde3ceace01f9d4c0a5e0a28ec79f3f091bd75c93595580d2e8bf5"} err="failed to get container status \"e35960aca8bde3ceace01f9d4c0a5e0a28ec79f3f091bd75c93595580d2e8bf5\": rpc error: code = NotFound desc = could not find container \"e35960aca8bde3ceace01f9d4c0a5e0a28ec79f3f091bd75c93595580d2e8bf5\": container with ID starting with e35960aca8bde3ceace01f9d4c0a5e0a28ec79f3f091bd75c93595580d2e8bf5 not found: ID does not exist" Jul 16 18:24:29.055276 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.055248 2582 scope.go:117] "RemoveContainer" containerID="092f13f82034a886a7356d5779b4f47ac0fbaaf6c000669b5abf1ed5e7bfb85a" Jul 16 18:24:29.062313 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.062291 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw"] Jul 16 18:24:29.063025 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.063010 2582 scope.go:117] "RemoveContainer" containerID="df4c5ea879819ab52f3fb68cf925d5c6695ee72bab4365b89becb372dc044ef6" Jul 16 18:24:29.065749 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.065721 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-97269-predictor-b796c5fd5-tzshw"] Jul 16 18:24:29.069987 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.069971 2582 scope.go:117] "RemoveContainer" containerID="092f13f82034a886a7356d5779b4f47ac0fbaaf6c000669b5abf1ed5e7bfb85a" Jul 16 18:24:29.070244 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:24:29.070225 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092f13f82034a886a7356d5779b4f47ac0fbaaf6c000669b5abf1ed5e7bfb85a\": container with ID starting with 092f13f82034a886a7356d5779b4f47ac0fbaaf6c000669b5abf1ed5e7bfb85a not found: ID does not exist" containerID="092f13f82034a886a7356d5779b4f47ac0fbaaf6c000669b5abf1ed5e7bfb85a" Jul 16 18:24:29.070308 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.070253 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092f13f82034a886a7356d5779b4f47ac0fbaaf6c000669b5abf1ed5e7bfb85a"} err="failed to get container status \"092f13f82034a886a7356d5779b4f47ac0fbaaf6c000669b5abf1ed5e7bfb85a\": rpc error: code = NotFound desc = could not find container \"092f13f82034a886a7356d5779b4f47ac0fbaaf6c000669b5abf1ed5e7bfb85a\": container with ID starting with 092f13f82034a886a7356d5779b4f47ac0fbaaf6c000669b5abf1ed5e7bfb85a not found: ID does not exist" Jul 16 18:24:29.070308 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.070269 2582 scope.go:117] "RemoveContainer" containerID="df4c5ea879819ab52f3fb68cf925d5c6695ee72bab4365b89becb372dc044ef6" Jul 16 18:24:29.070490 ip-10-0-139-103 kubenswrapper[2582]: E0716 18:24:29.070452 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df4c5ea879819ab52f3fb68cf925d5c6695ee72bab4365b89becb372dc044ef6\": container with ID starting with df4c5ea879819ab52f3fb68cf925d5c6695ee72bab4365b89becb372dc044ef6 not found: ID does not exist" containerID="df4c5ea879819ab52f3fb68cf925d5c6695ee72bab4365b89becb372dc044ef6" Jul 16 18:24:29.070536 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.070492 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4c5ea879819ab52f3fb68cf925d5c6695ee72bab4365b89becb372dc044ef6"} err="failed to get container status \"df4c5ea879819ab52f3fb68cf925d5c6695ee72bab4365b89becb372dc044ef6\": rpc error: code = NotFound desc = could not find container \"df4c5ea879819ab52f3fb68cf925d5c6695ee72bab4365b89becb372dc044ef6\": container with ID starting with df4c5ea879819ab52f3fb68cf925d5c6695ee72bab4365b89becb372dc044ef6 not found: ID does not exist" Jul 16 18:24:29.109904 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.109886 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzv7z\" (UniqueName: \"kubernetes.io/projected/ae46f104-584c-44ce-ae08-507d364f8971-kube-api-access-rzv7z\") pod \"ae46f104-584c-44ce-ae08-507d364f8971\" (UID: \"ae46f104-584c-44ce-ae08-507d364f8971\") " Jul 16 18:24:29.109993 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.109948 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae46f104-584c-44ce-ae08-507d364f8971-kube-rbac-proxy-sar-config\") pod \"ae46f104-584c-44ce-ae08-507d364f8971\" (UID: \"ae46f104-584c-44ce-ae08-507d364f8971\") " Jul 16 18:24:29.109993 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.109976 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae46f104-584c-44ce-ae08-507d364f8971-proxy-tls\") pod \"ae46f104-584c-44ce-ae08-507d364f8971\" (UID: \"ae46f104-584c-44ce-ae08-507d364f8971\") " Jul 16 18:24:29.110199 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.110073 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cb9m\" (UniqueName: \"kubernetes.io/projected/ef32f086-22fa-4544-a6e3-b6e27db85bfc-kube-api-access-6cb9m\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:24:29.110199 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.110083 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef32f086-22fa-4544-a6e3-b6e27db85bfc-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:24:29.110199 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.110091 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef32f086-22fa-4544-a6e3-b6e27db85bfc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:24:29.110366 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.110319 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae46f104-584c-44ce-ae08-507d364f8971-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae46f104-584c-44ce-ae08-507d364f8971" (UID: "ae46f104-584c-44ce-ae08-507d364f8971"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:24:29.111920 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.111898 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae46f104-584c-44ce-ae08-507d364f8971-kube-api-access-rzv7z" (OuterVolumeSpecName: "kube-api-access-rzv7z") pod "ae46f104-584c-44ce-ae08-507d364f8971" (UID: "ae46f104-584c-44ce-ae08-507d364f8971"). InnerVolumeSpecName "kube-api-access-rzv7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 18:24:29.111981 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.111964 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae46f104-584c-44ce-ae08-507d364f8971-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae46f104-584c-44ce-ae08-507d364f8971" (UID: "ae46f104-584c-44ce-ae08-507d364f8971"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:24:29.211260 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.211196 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae46f104-584c-44ce-ae08-507d364f8971-proxy-tls\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:24:29.211260 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.211228 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzv7z\" (UniqueName: \"kubernetes.io/projected/ae46f104-584c-44ce-ae08-507d364f8971-kube-api-access-rzv7z\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:24:29.211260 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.211237 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae46f104-584c-44ce-ae08-507d364f8971-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-103.ec2.internal\" DevicePath \"\"" Jul 16 18:24:29.362610 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.362576 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g"] Jul 16 18:24:29.367030 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:29.367010 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-97269-predictor-7d7f6f997b-9t55g"] Jul 16 18:24:30.583980 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:30.583945 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae46f104-584c-44ce-ae08-507d364f8971" path="/var/lib/kubelet/pods/ae46f104-584c-44ce-ae08-507d364f8971/volumes" Jul 16 18:24:30.584347 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:30.584331 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" path="/var/lib/kubelet/pods/ef32f086-22fa-4544-a6e3-b6e27db85bfc/volumes" Jul 16 18:24:59.237144 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237110 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hpk2v/must-gather-hfpb8"] Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237364 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237374 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237383 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237389 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237396 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237402 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237408 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237413 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237422 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237427 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237435 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237440 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237446 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237451 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237456 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237461 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237482 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237488 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237494 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237499 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237505 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237511 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237521 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237526 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237567 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237575 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237580 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237586 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237591 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2f85123-b57b-4560-b655-029db7a3ce32" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237595 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237599 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2817fb1c-1f8c-4eda-86be-79a4078085ab" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237605 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237609 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef32f086-22fa-4544-a6e3-b6e27db85bfc" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237615 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae46f104-584c-44ce-ae08-507d364f8971" containerName="kube-rbac-proxy" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237621 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e68e74-6ad3-43ac-bd1d-dabe89c07510" containerName="kserve-container" Jul 16 18:24:59.237630 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.237626 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8f53a82-a7e5-4703-9241-d93ff28b09c6" containerName="kube-rbac-proxy" Jul 16 18:24:59.240449 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.240431 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hpk2v/must-gather-hfpb8" Jul 16 18:24:59.242950 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.242928 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hpk2v\"/\"openshift-service-ca.crt\"" Jul 16 18:24:59.243828 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.243810 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hpk2v\"/\"kube-root-ca.crt\"" Jul 16 18:24:59.243926 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.243856 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hpk2v\"/\"default-dockercfg-g2x2f\"" Jul 16 18:24:59.249061 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.249031 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hpk2v/must-gather-hfpb8"] Jul 16 18:24:59.419392 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.419354 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqf2h\" (UniqueName: \"kubernetes.io/projected/1d59f413-2487-43ac-aa78-b62b16e017b4-kube-api-access-kqf2h\") pod \"must-gather-hfpb8\" (UID: \"1d59f413-2487-43ac-aa78-b62b16e017b4\") " pod="openshift-must-gather-hpk2v/must-gather-hfpb8" Jul 16 18:24:59.419392 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.419396 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d59f413-2487-43ac-aa78-b62b16e017b4-must-gather-output\") pod \"must-gather-hfpb8\" (UID: \"1d59f413-2487-43ac-aa78-b62b16e017b4\") " pod="openshift-must-gather-hpk2v/must-gather-hfpb8" Jul 16 18:24:59.520723 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.520659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d59f413-2487-43ac-aa78-b62b16e017b4-must-gather-output\") pod \"must-gather-hfpb8\" (UID: \"1d59f413-2487-43ac-aa78-b62b16e017b4\") " pod="openshift-must-gather-hpk2v/must-gather-hfpb8" Jul 16 18:24:59.520911 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.520746 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqf2h\" (UniqueName: \"kubernetes.io/projected/1d59f413-2487-43ac-aa78-b62b16e017b4-kube-api-access-kqf2h\") pod \"must-gather-hfpb8\" (UID: \"1d59f413-2487-43ac-aa78-b62b16e017b4\") " pod="openshift-must-gather-hpk2v/must-gather-hfpb8" Jul 16 18:24:59.520995 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.520966 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d59f413-2487-43ac-aa78-b62b16e017b4-must-gather-output\") pod \"must-gather-hfpb8\" (UID: \"1d59f413-2487-43ac-aa78-b62b16e017b4\") " pod="openshift-must-gather-hpk2v/must-gather-hfpb8" Jul 16 18:24:59.530411 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.530385 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqf2h\" (UniqueName: \"kubernetes.io/projected/1d59f413-2487-43ac-aa78-b62b16e017b4-kube-api-access-kqf2h\") pod \"must-gather-hfpb8\" (UID: \"1d59f413-2487-43ac-aa78-b62b16e017b4\") " pod="openshift-must-gather-hpk2v/must-gather-hfpb8" Jul 16 18:24:59.550217 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.550197 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hpk2v/must-gather-hfpb8" Jul 16 18:24:59.677447 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.677418 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hpk2v/must-gather-hfpb8"] Jul 16 18:24:59.681031 ip-10-0-139-103 kubenswrapper[2582]: W0716 18:24:59.681005 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d59f413_2487_43ac_aa78_b62b16e017b4.slice/crio-73e013fdeee8c98b1d10a8726060121100b0aa8e2270ecf1a097c1ed8c026a14 WatchSource:0}: Error finding container 73e013fdeee8c98b1d10a8726060121100b0aa8e2270ecf1a097c1ed8c026a14: Status 404 returned error can't find the container with id 73e013fdeee8c98b1d10a8726060121100b0aa8e2270ecf1a097c1ed8c026a14 Jul 16 18:24:59.683345 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:24:59.683326 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 18:25:00.154161 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:00.154126 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpk2v/must-gather-hfpb8" event={"ID":"1d59f413-2487-43ac-aa78-b62b16e017b4","Type":"ContainerStarted","Data":"73e013fdeee8c98b1d10a8726060121100b0aa8e2270ecf1a097c1ed8c026a14"} Jul 16 18:25:01.162373 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:01.162326 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpk2v/must-gather-hfpb8" event={"ID":"1d59f413-2487-43ac-aa78-b62b16e017b4","Type":"ContainerStarted","Data":"c529c6184da22e643e0b67ed825db2c4c433cfaf6a6aeee55d4a615c95f0efce"} Jul 16 18:25:01.162867 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:01.162409 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpk2v/must-gather-hfpb8" event={"ID":"1d59f413-2487-43ac-aa78-b62b16e017b4","Type":"ContainerStarted","Data":"f0ed7fe145de9ae0833609d8b35adc0876ac280184a54c2874620893ea61ddbe"} Jul 16 18:25:02.025671 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:02.025633 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hfrlv_503d02a0-3e8a-4786-bf7e-c4c0bec8dbda/global-pull-secret-syncer/0.log" Jul 16 18:25:02.141957 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:02.141925 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7kc7q_30219802-706a-47e2-9bf1-8942aade426c/konnectivity-agent/0.log" Jul 16 18:25:02.234050 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:02.234015 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-103.ec2.internal_b8cc93a3e3b0301bf02b2011bdc86d70/haproxy/0.log" Jul 16 18:25:06.507800 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:06.507750 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-bgl8d_b0ec8f32-c15f-4cdb-aefe-c0c197ba1ca0/cluster-monitoring-operator/0.log" Jul 16 18:25:06.862997 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:06.862909 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vqkkm_44374839-36f1-4dcd-8e22-3280caf48704/node-exporter/0.log" Jul 16 18:25:06.888010 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:06.887971 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vqkkm_44374839-36f1-4dcd-8e22-3280caf48704/kube-rbac-proxy/0.log" Jul 16 18:25:06.922882 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:06.922853 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vqkkm_44374839-36f1-4dcd-8e22-3280caf48704/init-textfile/0.log" Jul 16 18:25:08.805798 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:08.805740 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-k4gc5_edd90096-a7e6-4040-a60c-76b5030fe275/networking-console-plugin/0.log" Jul 16 18:25:09.255141 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.255101 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/1.log" Jul 16 18:25:09.261504 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.261462 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-2wxlk_c223b307-534a-47e4-b677-e015ec5e6861/console-operator/2.log" Jul 16 18:25:09.669208 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.669088 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hpk2v/must-gather-hfpb8" podStartSLOduration=9.881347045 podStartE2EDuration="10.669065419s" totalImagesPullingTime="787.718374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:25:01.178608635 +0000 UTC m=+3051.195837978" watchObservedRunningTime="2026-07-16 18:25:09.669065419 +0000 UTC m=+3059.686294765" Jul 16 18:25:09.669392 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.669333 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5"] Jul 16 18:25:09.674229 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.674207 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.682875 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.682853 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5"] Jul 16 18:25:09.710104 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.709047 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-lib-modules\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.710104 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.709088 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-sys\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.710104 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.709132 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss5jq\" (UniqueName: \"kubernetes.io/projected/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-kube-api-access-ss5jq\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.710104 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.709156 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-podres\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.710104 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.709223 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-proc\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.810657 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.810617 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-proc\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.811225 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.811201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-lib-modules\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.811356 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.811339 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-sys\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.811526 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.811505 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss5jq\" (UniqueName: \"kubernetes.io/projected/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-kube-api-access-ss5jq\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.811653 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.811639 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-podres\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.811733 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.811707 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-proc\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.811803 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.811670 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-sys\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.811803 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.811692 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-lib-modules\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.811803 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.811786 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-podres\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.821656 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.821627 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss5jq\" (UniqueName: \"kubernetes.io/projected/f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce-kube-api-access-ss5jq\") pod \"perf-node-gather-daemonset-45rf5\" (UID: \"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:09.985653 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:09.985608 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:10.147059 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:10.147026 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5"] Jul 16 18:25:10.150816 ip-10-0-139-103 kubenswrapper[2582]: W0716 18:25:10.150778 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf44a844b_3120_4bb7_a6ce_0fb8ee3c06ce.slice/crio-b1b59a4fbee3a5da16d7673fa608d62e7e5bc9fb56d367188773164e3e86c5b2 WatchSource:0}: Error finding container b1b59a4fbee3a5da16d7673fa608d62e7e5bc9fb56d367188773164e3e86c5b2: Status 404 returned error can't find the container with id b1b59a4fbee3a5da16d7673fa608d62e7e5bc9fb56d367188773164e3e86c5b2 Jul 16 18:25:10.210797 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:10.210764 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" event={"ID":"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce","Type":"ContainerStarted","Data":"b1b59a4fbee3a5da16d7673fa608d62e7e5bc9fb56d367188773164e3e86c5b2"} Jul 16 18:25:10.270372 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:10.270291 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-tmzwf_a724e2d9-c8a0-4cbf-b79b-3463893c6019/volume-data-source-validator/0.log" Jul 16 18:25:11.203382 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:11.203309 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-85spg_f59e0fc2-4364-4718-9c16-34bf2c019278/dns/0.log" Jul 16 18:25:11.216091 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:11.216050 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" event={"ID":"f44a844b-3120-4bb7-a6ce-0fb8ee3c06ce","Type":"ContainerStarted","Data":"886af8701f79b6b8d1a58ccf891d07c4bfa9b40f18eeff4a6fa17431a3e21c90"} Jul 16 18:25:11.216258 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:11.216169 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:11.227219 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:11.227183 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-85spg_f59e0fc2-4364-4718-9c16-34bf2c019278/kube-rbac-proxy/0.log" Jul 16 18:25:11.234698 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:11.234660 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" podStartSLOduration=2.234645359 podStartE2EDuration="2.234645359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:25:11.232914494 +0000 UTC m=+3061.250143855" watchObservedRunningTime="2026-07-16 18:25:11.234645359 +0000 UTC m=+3061.251874700" Jul 16 18:25:11.438331 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:11.438293 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d7b2n_782eb993-d0db-4c25-b990-34dbe4d7a5b9/dns-node-resolver/0.log" Jul 16 18:25:11.981671 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:11.981641 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4ptfz_5fa87729-4cd3-4453-a444-9c59a6514753/node-ca/0.log" Jul 16 18:25:13.187119 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:13.187092 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5v4vt_142b45fe-5371-4dc2-998b-a4aedafa976c/serve-healthcheck-canary/0.log" Jul 16 18:25:13.642319 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:13.642289 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hsd9q_c05683ac-5e47-45a9-8de4-90bcb06edc11/kube-rbac-proxy/0.log" Jul 16 18:25:13.664417 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:13.664391 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hsd9q_c05683ac-5e47-45a9-8de4-90bcb06edc11/exporter/0.log" Jul 16 18:25:13.686219 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:13.686194 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hsd9q_c05683ac-5e47-45a9-8de4-90bcb06edc11/extractor/0.log" Jul 16 18:25:15.779350 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:15.779322 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-576d7b7757-5rqdx_0ee5d930-4121-453b-847d-fb6edb7ba63f/manager/0.log" Jul 16 18:25:15.827553 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:15.827532 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-vmwwk_d80d490e-605e-4b33-9338-304ed98b3c60/server/0.log" Jul 16 18:25:17.234827 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:17.234789 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-45rf5" Jul 16 18:25:20.245166 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:20.245138 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-kv4hq_63681960-82a8-45eb-b340-adfa536295c3/migrator/0.log" Jul 16 18:25:20.273411 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:20.273386 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-kv4hq_63681960-82a8-45eb-b340-adfa536295c3/graceful-termination/0.log" Jul 16 18:25:21.731651 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:21.731623 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct6j5_63aefcda-40ba-4e5d-8fd4-491a9bf5b45b/kube-multus-additional-cni-plugins/0.log" Jul 16 18:25:21.752008 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:21.751988 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct6j5_63aefcda-40ba-4e5d-8fd4-491a9bf5b45b/egress-router-binary-copy/0.log" Jul 16 18:25:21.773870 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:21.773850 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct6j5_63aefcda-40ba-4e5d-8fd4-491a9bf5b45b/cni-plugins/0.log" Jul 16 18:25:21.797097 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:21.797068 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct6j5_63aefcda-40ba-4e5d-8fd4-491a9bf5b45b/bond-cni-plugin/0.log" Jul 16 18:25:21.820231 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:21.820209 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct6j5_63aefcda-40ba-4e5d-8fd4-491a9bf5b45b/routeoverride-cni/0.log" Jul 16 18:25:21.843874 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:21.843852 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct6j5_63aefcda-40ba-4e5d-8fd4-491a9bf5b45b/whereabouts-cni-bincopy/0.log" Jul 16 18:25:21.867140 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:21.867118 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct6j5_63aefcda-40ba-4e5d-8fd4-491a9bf5b45b/whereabouts-cni/0.log" Jul 16 18:25:22.162242 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:22.162175 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qj6ls_e59c2366-fc5e-4e31-a94f-5a39c99dc3d5/kube-multus/0.log" Jul 16 18:25:22.185006 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:22.184979 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5qrcw_dc349e33-d51c-40bf-8890-392c3393299d/network-metrics-daemon/0.log" Jul 16 18:25:22.207273 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:22.207250 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5qrcw_dc349e33-d51c-40bf-8890-392c3393299d/kube-rbac-proxy/0.log" Jul 16 18:25:23.047791 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:23.047758 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-controller/0.log" Jul 16 18:25:23.066433 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:23.066408 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/1.log" Jul 16 18:25:23.066606 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:23.066493 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovn-acl-logging/0.log" Jul 16 18:25:23.086238 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:23.086215 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/kube-rbac-proxy-node/0.log" Jul 16 18:25:23.186807 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:23.186783 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/kube-rbac-proxy-ovn-metrics/0.log" Jul 16 18:25:23.205704 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:23.205679 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/northd/0.log" Jul 16 18:25:23.226806 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:23.226786 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/nbdb/0.log" Jul 16 18:25:23.248817 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:23.248792 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/sbdb/0.log" Jul 16 18:25:23.304605 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:23.304553 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4lfqt_807cdaf2-aa32-4788-9661-fb1b56a34fc2/ovnkube-controller/0.log" Jul 16 18:25:24.753148 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:24.753120 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-x2vn2_458ea307-1c97-4cda-8723-97e047f76a50/network-check-target-container/0.log" Jul 16 18:25:25.687179 ip-10-0-139-103 kubenswrapper[2582]: I0716 18:25:25.687154 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-855sl_e4e92fef-28b7-4d9e-af27-d22d54ca66a6/iptables-alerter/0.log"