Jul 06 20:33:26.835468 ip-10-0-142-142 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 20:33:26.835481 ip-10-0-142-142 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 20:33:26.835489 ip-10-0-142-142 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 20:33:26.835801 ip-10-0-142-142 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 20:33:36.899222 ip-10-0-142-142 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 20:33:36.899240 ip-10-0-142-142 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d21067e509be475bbc809a91eaea4498 -- Jul 06 20:35:32.795070 ip-10-0-142-142 systemd[1]: Starting Kubernetes Kubelet... Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.232942 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239660 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239673 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239678 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239685 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239690 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239694 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239698 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:35:33.269795 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239702 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239706 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239710 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239713 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239717 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239721 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239724 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239734 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239738 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239741 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239745 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239749 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239753 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239769 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239773 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239777 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239780 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239784 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239788 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239792 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:35:33.271342 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239796 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239800 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239803 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239807 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239810 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239814 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239817 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239821 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239825 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239828 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239832 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239835 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239838 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239843 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239846 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239852 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239857 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239861 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239865 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:35:33.271921 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239869 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239872 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239877 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239881 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239886 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239890 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239895 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239899 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239903 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239906 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239911 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239914 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239918 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239923 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239927 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239931 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239935 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239938 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239942 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:35:33.411325 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239946 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:35:33.307024 ip-10-0-142-142 systemd[1]: Started Kubernetes Kubelet. Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239950 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239953 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239957 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239961 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239965 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239969 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239972 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239977 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239981 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239984 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239987 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239991 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239996 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.239999 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240004 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240009 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240012 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240016 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240021 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:35:33.412138 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240024 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240028 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240031 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240034 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240038 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240041 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240045 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240049 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240052 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240055 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240059 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240063 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240067 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.240071 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241113 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241123 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241128 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241132 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241137 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241141 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:35:33.412894 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241144 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241148 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241153 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241157 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241160 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241165 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241169 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241173 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241177 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241181 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241185 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241189 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241192 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241196 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241200 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241206 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241212 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241216 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241220 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241224 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:35:33.413572 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241228 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241233 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241238 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241242 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241246 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241250 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241254 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241259 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241263 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241266 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241270 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241274 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241277 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241281 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241285 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241289 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241292 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241296 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241300 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:35:33.414177 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241303 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241307 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241311 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241314 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241320 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241325 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241328 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241332 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241335 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241339 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241343 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241346 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241350 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241353 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241356 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241360 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241364 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241368 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241372 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:35:33.414781 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241385 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241390 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241394 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241398 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241401 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241406 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241409 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241414 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241417 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241421 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241424 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241428 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241433 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241436 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241441 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241444 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241448 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241452 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241456 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241459 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:35:33.415323 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241463 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241466 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241470 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241474 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241477 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241481 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241485 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241488 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241491 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241495 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241499 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241503 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241507 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241511 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.241515 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242022 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242029 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242033 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242038 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242041 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:35:33.415950 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242045 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242051 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242056 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242060 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242064 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242067 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242071 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242076 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242079 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242083 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242087 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242091 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242095 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242099 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242103 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242106 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242110 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242113 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242117 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242121 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:35:33.416542 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242124 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242127 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242131 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242134 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242138 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242143 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242147 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242150 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242154 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242158 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242161 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242165 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242168 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242172 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242175 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242179 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242182 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242188 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242192 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:35:33.417194 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242196 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242203 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242207 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242210 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242214 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242217 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242221 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242225 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242228 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242232 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242236 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242239 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242243 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242246 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242250 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242254 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242259 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242263 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242266 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242270 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:35:33.417822 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242274 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242277 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242282 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242285 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242289 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242293 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242298 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242303 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242307 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242311 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242315 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242319 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242323 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242328 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242332 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242336 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242340 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242343 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242347 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:35:33.418611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242350 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242354 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242358 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242361 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242365 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242368 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242373 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242377 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242381 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242386 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242389 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242393 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242397 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242402 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242405 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.242409 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242493 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242501 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242509 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242515 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242520 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 20:35:33.419229 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242525 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242530 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242536 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242540 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242545 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242550 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242555 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242560 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242564 2582 flags.go:64] FLAG: --cgroup-root="" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242569 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242573 2582 flags.go:64] FLAG: --client-ca-file="" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242577 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242582 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242588 2582 flags.go:64] FLAG: --cluster-domain="" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242592 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242597 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242601 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242605 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242610 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242614 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242620 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242625 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242628 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242632 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 20:35:33.419990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242637 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242642 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242645 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242651 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242655 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242659 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242663 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242667 2582 flags.go:64] FLAG: --enable-server="true" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242670 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242677 2582 flags.go:64] FLAG: --event-burst="100" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242681 2582 flags.go:64] FLAG: --event-qps="50" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242684 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242689 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242695 2582 flags.go:64] FLAG: --eviction-hard="" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242700 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242704 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242708 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242713 2582 flags.go:64] FLAG: --eviction-soft="" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242717 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242721 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242725 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242729 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242733 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242737 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242741 2582 flags.go:64] FLAG: --feature-gates="" Jul 06 20:35:33.420574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242746 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242751 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242755 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242775 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242779 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242783 2582 flags.go:64] FLAG: --help="false" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242788 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242792 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242797 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242802 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242806 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242811 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242815 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242819 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242823 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242827 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242831 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242835 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242839 2582 flags.go:64] FLAG: --kube-reserved="" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242843 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242849 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242853 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242857 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242862 2582 flags.go:64] FLAG: --lock-file="" Jul 06 20:35:33.421315 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242866 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242871 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242875 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242882 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242886 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242890 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242894 2582 flags.go:64] FLAG: --logging-format="text" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242897 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242902 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242906 2582 flags.go:64] FLAG: --manifest-url="" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242910 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242917 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242921 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242927 2582 flags.go:64] FLAG: --max-pods="110" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242931 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242934 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242938 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242943 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242948 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242953 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242957 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242966 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242970 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242975 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 20:35:33.421952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242979 2582 flags.go:64] FLAG: --pod-cidr="" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242983 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242990 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242994 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.242998 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243004 2582 flags.go:64] FLAG: --port="10250" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243008 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243011 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0929eb19b8af4a509" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243016 2582 flags.go:64] FLAG: --qos-reserved="" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243021 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243026 2582 flags.go:64] FLAG: --register-node="true" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243030 2582 flags.go:64] FLAG: --register-with-taints="" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243034 2582 flags.go:64] FLAG: --registry-burst="10" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243038 2582 flags.go:64] FLAG: --registry-qps="5" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243042 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243046 2582 flags.go:64] FLAG: --reserved-memory="" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243051 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243055 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243063 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243069 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243073 2582 flags.go:64] FLAG: --runonce="false" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243077 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243080 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243084 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243089 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 20:35:33.422593 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243093 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243097 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243102 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243106 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243110 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243114 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243118 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243122 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243127 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243130 2582 flags.go:64] FLAG: --system-cgroups="" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243134 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243142 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243148 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243151 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243157 2582 flags.go:64] FLAG: --tls-min-version="" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243161 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243165 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243169 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243173 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243177 2582 flags.go:64] FLAG: --v="2" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243183 2582 flags.go:64] FLAG: --version="false" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243188 2582 flags.go:64] FLAG: --vmodule="" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243193 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.243198 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243367 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:35:33.423641 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243372 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243378 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243383 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243387 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243392 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243395 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243400 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243403 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243407 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243411 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243415 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243419 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243423 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243427 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243431 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243435 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243438 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243441 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243445 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243451 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:35:33.424338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243454 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243458 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243462 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243466 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243469 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243473 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243477 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243481 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243485 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243489 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243493 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243497 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243500 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243505 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243508 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243512 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243516 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243520 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243523 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243527 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:35:33.638827 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243531 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243535 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243539 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243545 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243550 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243554 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243558 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243562 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243565 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243569 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243573 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243581 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243585 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243589 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243594 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243598 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243601 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243605 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243609 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:35:33.659819 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243613 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243618 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243621 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243625 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243628 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243632 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243637 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243640 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243644 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243648 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243652 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243656 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243659 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243662 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243666 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243670 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243673 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243678 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243681 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243685 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:35:33.660818 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243689 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243692 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243696 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243699 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243704 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243708 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243712 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243716 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243719 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243723 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243726 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243730 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243734 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243737 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243741 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243745 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243748 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243752 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:35:33.661510 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.243772 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.244373 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.255414 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.255429 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.255448 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.255453 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255512 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255517 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255520 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255523 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255526 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255529 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255532 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255535 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255538 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:35:33.662112 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255541 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255543 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255546 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255549 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255553 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255557 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255561 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255566 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255568 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255571 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255574 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255576 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255580 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255582 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255585 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255593 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255597 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255599 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255602 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255605 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:35:33.662611 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255608 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255611 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255614 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255616 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255619 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255621 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255624 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255626 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255629 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255633 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255637 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255641 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255647 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255652 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255654 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255658 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255662 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255665 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255668 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:35:33.663275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255671 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255674 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255676 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255680 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255684 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255686 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255689 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255692 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255697 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255700 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255703 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255705 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255710 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255719 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255723 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255725 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255728 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255730 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255733 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:35:33.663897 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255736 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255738 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255741 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255744 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255746 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255748 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255751 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255754 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255771 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255776 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255780 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255784 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255789 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255792 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255794 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255797 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255799 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255802 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255804 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255807 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:35:33.664507 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255809 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255813 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255816 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255818 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255821 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255824 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255826 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255829 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255831 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255834 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255836 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255839 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.255844 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255979 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255985 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:35:33.665338 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255988 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255991 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255993 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.255999 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256002 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256005 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256008 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256013 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256017 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256022 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256027 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256029 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256032 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256034 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256037 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256039 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256042 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256044 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:35:33.666232 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256047 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256052 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256056 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256059 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256061 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256064 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256067 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256069 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256071 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256074 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256076 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256079 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256081 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256084 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256087 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256091 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256095 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256101 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256104 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256107 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:35:33.667122 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256109 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256111 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256115 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256119 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256121 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256125 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256128 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256130 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256133 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256135 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256137 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256140 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256142 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256147 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256149 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256151 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256154 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256157 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256159 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:35:33.667767 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256162 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256164 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256168 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256172 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256176 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256180 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256183 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256186 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256188 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256190 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256194 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256197 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256199 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256202 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256204 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256206 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256209 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256212 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256214 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256217 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:35:33.668423 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256220 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256223 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256225 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256227 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256230 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256232 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256236 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256238 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256241 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256243 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256247 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256251 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256255 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256259 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256262 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256265 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256267 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256269 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256273 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:35:33.669033 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:33.256276 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:35:33.669613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.256281 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:35:33.669613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.256381 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 20:35:33.669613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.258487 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 20:35:33.669613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.259772 2582 server.go:1019] "Starting client certificate rotation" Jul 06 20:35:33.669613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.259865 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 20:35:33.669613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.259895 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 20:35:33.669613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.262457 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 20:35:33.669613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.263557 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 20:35:33.669613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.274936 2582 log.go:25] "Validated CRI v1 runtime API" Jul 06 20:35:33.669613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.277420 2582 log.go:25] "Validated CRI v1 image API" Jul 06 20:35:33.669613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.278203 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 20:35:33.669613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.280790 2582 fs.go:135] Filesystem UUIDs: map[681c076c-5b16-46b3-94c5-da3616cfe69f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bdfc231e-04f4-447b-8618-f41feebae231:/dev/nvme0n1p3] Jul 06 20:35:33.670012 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.280804 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 20:35:33.670127 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.285383 2582 manager.go:216] Machine: {Timestamp:2026-07-06 20:35:33.284920416 +0000 UTC m=+0.370475599 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199773 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d07bb800794387a6c10a6a6df5f9f SystemUUID:ec2d07bb-8007-9438-7a6c-10a6a6df5f9f BootID:d21067e5-09be-475b-bc80-9a91eaea4498 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4e:2b:38:73:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4e:2b:38:73:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:f6:1c:12:39:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 20:35:33.670127 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.285509 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 20:35:33.670127 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.285602 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 20:35:33.670127 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.288410 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 20:35:33.670127 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.288453 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 20:35:33.670127 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.288611 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 20:35:33.670127 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.289855 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.289868 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.289876 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.290468 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.291654 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.294824 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.294932 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.294957 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.294965 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.294975 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.294985 2582 kubelet.go:398] "Adding apiserver pod source" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.294996 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.295883 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.295900 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.298048 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.299886 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.299912 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.301108 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.301121 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.301126 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.301131 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.301139 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.301144 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.301151 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.301157 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.301163 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.301169 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.301178 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.301183 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.302486 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.302496 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.305361 2582 server.go:1268] "Started kubelet" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.305478 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.305526 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.305577 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.305976 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.306829 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.306829 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.307382 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 20:35:33.670397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.308779 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.311134 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.314201 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.314386 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-142.ec2.internal\" not found" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.313811 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-142.ec2.internal.18bfcdb002b4974c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-142.ec2.internal,UID:ip-10-0-142-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-142.ec2.internal,},FirstTimestamp:2026-07-06 20:35:33.30533358 +0000 UTC m=+0.390888764,LastTimestamp:2026-07-06 20:35:33.30533358 +0000 UTC m=+0.390888764,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-142.ec2.internal,}" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.314882 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.314892 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.314979 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.315056 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.315062 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.315367 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.315600 2582 factory.go:55] Registering systemd factory Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.315613 2582 factory.go:223] Registration of the systemd container factory successfully Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.315803 2582 factory.go:153] Registering CRI-O factory Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.315814 2582 factory.go:223] Registration of the crio container factory successfully Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.315871 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.315905 2582 factory.go:103] Registering Raw factory Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.315918 2582 manager.go:1196] Started watching for new ooms in manager Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.316280 2582 manager.go:349] Starting recovery of all containers Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.319509 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.319575 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.329629 2582 manager.go:354] Recovery completed Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.338453 2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.345114 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.347006 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.347037 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.347052 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.347533 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.348248 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.348270 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:33.671521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.348282 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.348960 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-142.ec2.internal.18bfcdb00530b081 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-142.ec2.internal,UID:ip-10-0-142-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-142.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-142.ec2.internal,},FirstTimestamp:2026-07-06 20:35:33.347020929 +0000 UTC m=+0.432576118,LastTimestamp:2026-07-06 20:35:33.347020929 +0000 UTC m=+0.432576118,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-142.ec2.internal,}" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.349688 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.349700 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.349715 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.351117 2582 policy_none.go:49] "None policy: Start" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.351130 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.351139 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.352171 2582 policy_none.go:47] "Start" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.357843 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tqw82" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.358818 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-142.ec2.internal.18bfcdb005310b0e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-142.ec2.internal,UID:ip-10-0-142-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-142.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-142.ec2.internal,},FirstTimestamp:2026-07-06 20:35:33.34704411 +0000 UTC m=+0.432599295,LastTimestamp:2026-07-06 20:35:33.34704411 +0000 UTC m=+0.432599295,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-142.ec2.internal,}" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.367342 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tqw82" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.368234 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-142.ec2.internal.18bfcdb005313e97 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-142.ec2.internal,UID:ip-10-0-142-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-142.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-142.ec2.internal,},FirstTimestamp:2026-07-06 20:35:33.347057303 +0000 UTC m=+0.432612487,LastTimestamp:2026-07-06 20:35:33.347057303 +0000 UTC m=+0.432612487,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-142.ec2.internal,}" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.387303 2582 manager.go:337] "Starting Device Plugin manager" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.387430 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.387439 2582 server.go:87] "Starting device plugin registration server" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.387626 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.387636 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.387740 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.387819 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.387829 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 20:35:33.672720 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.388403 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.388447 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-142.ec2.internal\" not found" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.487974 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.488885 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.488911 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.488920 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.488944 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.502092 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.502111 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-142.ec2.internal\": node \"ip-10-0-142-142.ec2.internal\" not found" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.529963 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.531168 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.531205 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.531235 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.531269 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.535443 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.547537 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-142.ec2.internal\" not found" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.631864 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-142.ec2.internal"] Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.631947 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.632857 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.632890 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.632901 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.634219 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.634399 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.634434 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.638973 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.638998 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.639012 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.639049 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.639072 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.639081 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.640711 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.640736 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.641481 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.641509 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:35:33.853937 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.641523 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.647933 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-142.ec2.internal\" not found" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.676684 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-142.ec2.internal\" not found" node="ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.681488 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-142.ec2.internal\" not found" node="ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.716670 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/900a8e8020660ce714cfb40f4b668905-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal\" (UID: \"900a8e8020660ce714cfb40f4b668905\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.716715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/900a8e8020660ce714cfb40f4b668905-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal\" (UID: \"900a8e8020660ce714cfb40f4b668905\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.716740 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2150a52d3ba57389650b4becb0fa9935-config\") pod \"kube-apiserver-proxy-ip-10-0-142-142.ec2.internal\" (UID: \"2150a52d3ba57389650b4becb0fa9935\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.748818 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-142.ec2.internal\" not found" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.817260 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/900a8e8020660ce714cfb40f4b668905-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal\" (UID: \"900a8e8020660ce714cfb40f4b668905\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.817290 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/900a8e8020660ce714cfb40f4b668905-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal\" (UID: \"900a8e8020660ce714cfb40f4b668905\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.817308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2150a52d3ba57389650b4becb0fa9935-config\") pod \"kube-apiserver-proxy-ip-10-0-142-142.ec2.internal\" (UID: \"2150a52d3ba57389650b4becb0fa9935\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.817336 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2150a52d3ba57389650b4becb0fa9935-config\") pod \"kube-apiserver-proxy-ip-10-0-142-142.ec2.internal\" (UID: \"2150a52d3ba57389650b4becb0fa9935\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.817362 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/900a8e8020660ce714cfb40f4b668905-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal\" (UID: \"900a8e8020660ce714cfb40f4b668905\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.817367 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/900a8e8020660ce714cfb40f4b668905-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal\" (UID: \"900a8e8020660ce714cfb40f4b668905\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal" Jul 06 20:35:33.855986 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.849006 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-142.ec2.internal\" not found" Jul 06 20:35:34.022375 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:33.950863 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-142.ec2.internal\" not found" Jul 06 20:35:34.022375 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.981299 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal" Jul 06 20:35:34.022375 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:33.984664 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-142.ec2.internal" Jul 06 20:35:34.231701 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.051653 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-142.ec2.internal\" not found" Jul 06 20:35:34.231701 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.152083 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-142.ec2.internal\" not found" Jul 06 20:35:34.231701 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.217243 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.260379 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.260503 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.260551 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.297449 2582 apiserver.go:52] "Watching apiserver" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.319857 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.319855 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.324822 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.325128 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg","openshift-multus/multus-additional-cni-plugins-jqtv5","openshift-network-operator/iptables-alerter-5jb4d","openshift-ovn-kubernetes/ovnkube-node-crszk","kube-system/global-pull-secret-syncer-5zpwl","openshift-cluster-node-tuning-operator/tuned-jwpp2","openshift-dns/node-resolver-nsbpk","openshift-image-registry/node-ca-fzclm","openshift-multus/multus-6jqmg","openshift-multus/network-metrics-daemon-d26rt","openshift-network-diagnostics/network-check-target-sxdx7","kube-system/konnectivity-agent-crf7d"] Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.327683 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.329909 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.330529 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qjz2g\"" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.330830 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.330833 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.331380 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.332066 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5jb4d" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.332543 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.332671 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.332803 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.332876 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.333061 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.333079 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rz2j7\"" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.334709 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.334771 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-142.ec2.internal" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.336748 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.337290 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.337442 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bnd65\"" Jul 06 20:35:34.421164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.337576 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.337801 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.337907 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.337979 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.339299 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.339611 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.340789 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gfwvs\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.340795 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.340972 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.340980 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.341049 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.341063 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.341560 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nsbpk" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.342054 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.342560 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5nls2\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.342775 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.342839 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.343721 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fzclm" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.344680 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.344945 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.345331 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-25dbw\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.346566 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.347235 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-spzvm\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.347291 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.347306 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.347362 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.347597 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.349755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:34.422438 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.349867 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.350726 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.350945 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-splnw\"" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.352512 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.352602 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.354162 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal"] Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.354188 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-142.ec2.internal"] Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.354261 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-crf7d" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.357695 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.357716 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.358141 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zs6kf\"" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.362798 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qxgg4" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.369844 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 20:30:34 +0000 UTC" deadline="2028-02-04 20:22:01.782131936 +0000 UTC" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.369888 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13871h46m27.412248032s" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.374653 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qxgg4" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.412915 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.415968 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423045 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-kubernetes\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423089 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-device-dir\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423115 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423137 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-var-lib-cni-multus\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423172 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d909edb1-81aa-4c46-8b90-18691df9680c-agent-certs\") pod \"konnectivity-agent-crf7d\" (UID: \"d909edb1-81aa-4c46-8b90-18691df9680c\") " pod="kube-system/konnectivity-agent-crf7d" Jul 06 20:35:34.424067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423197 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-cni-bin\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423219 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-var-lib-kubelet\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423245 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-run-k8s-cni-cncf-io\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423270 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423293 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-run-systemd\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423317 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-cni-netd\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423348 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423372 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-run\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423394 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9gmp\" (UniqueName: \"kubernetes.io/projected/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-kube-api-access-c9gmp\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423431 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-slash\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js5mb\" (UniqueName: \"kubernetes.io/projected/82ffacd0-c326-48d3-8e22-be96a8e06e17-kube-api-access-js5mb\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423491 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-host\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423517 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a203ceeb-e878-49c1-b1e1-ff93471aa9fb-host\") pod \"node-ca-fzclm\" (UID: \"a203ceeb-e878-49c1-b1e1-ff93471aa9fb\") " pod="openshift-image-registry/node-ca-fzclm" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423542 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tq6w\" (UniqueName: \"kubernetes.io/projected/a203ceeb-e878-49c1-b1e1-ff93471aa9fb-kube-api-access-7tq6w\") pod \"node-ca-fzclm\" (UID: \"a203ceeb-e878-49c1-b1e1-ff93471aa9fb\") " pod="openshift-image-registry/node-ca-fzclm" Jul 06 20:35:34.425163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423579 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-cnibin\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423606 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d909edb1-81aa-4c46-8b90-18691df9680c-konnectivity-ca\") pod \"konnectivity-agent-crf7d\" (UID: \"d909edb1-81aa-4c46-8b90-18691df9680c\") " pod="kube-system/konnectivity-agent-crf7d" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423625 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-var-lib-openvswitch\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-run-ovn\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423659 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-var-lib-kubelet\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423675 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwpk\" (UniqueName: \"kubernetes.io/projected/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-kube-api-access-plwpk\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423696 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-multus-conf-dir\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423720 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-run-netns\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423752 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-etc-openvswitch\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423810 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-run-ovn-kubernetes\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423846 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423879 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e609c626-c0c6-4b01-9640-e63a163c6c96-hosts-file\") pod \"node-resolver-nsbpk\" (UID: \"e609c626-c0c6-4b01-9640-e63a163c6c96\") " pod="openshift-dns/node-resolver-nsbpk" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423907 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-var-lib-cni-bin\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423935 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-cnibin\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.425873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.423961 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/82ffacd0-c326-48d3-8e22-be96a8e06e17-ovn-node-metrics-cert\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424003 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-sysconfig\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424028 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-lib-modules\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a203ceeb-e878-49c1-b1e1-ff93471aa9fb-serviceca\") pod \"node-ca-fzclm\" (UID: \"a203ceeb-e878-49c1-b1e1-ff93471aa9fb\") " pod="openshift-image-registry/node-ca-fzclm" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424082 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-system-cni-dir\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424109 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424143 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcsbh\" (UniqueName: \"kubernetes.io/projected/3268a9a6-5687-440f-91a0-5916a32aa1a5-kube-api-access-pcsbh\") pod \"iptables-alerter-5jb4d\" (UID: \"3268a9a6-5687-440f-91a0-5916a32aa1a5\") " pod="openshift-network-operator/iptables-alerter-5jb4d" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424169 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-kubelet\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424194 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-node-log\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424233 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-systemd\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424258 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424284 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-run-multus-certs\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424308 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-etc-kubernetes\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424335 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch6gp\" (UniqueName: \"kubernetes.io/projected/2a6c9a93-1c46-445b-a246-7251d7f007c9-kube-api-access-ch6gp\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.426367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424361 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63a48280-235d-4f85-a741-ff9d9c3aaa3b-kubelet-config\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63a48280-235d-4f85-a741-ff9d9c3aaa3b-dbus\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424471 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-modprobe-d\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424522 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-sysctl-conf\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424550 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-registration-dir\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424605 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-hostroot\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424699 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a6c9a93-1c46-445b-a246-7251d7f007c9-multus-daemon-config\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424725 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-systemd-units\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/82ffacd0-c326-48d3-8e22-be96a8e06e17-ovnkube-config\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424797 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424829 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-sys\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424852 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-sys-fs\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424872 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e609c626-c0c6-4b01-9640-e63a163c6c96-tmp-dir\") pod \"node-resolver-nsbpk\" (UID: \"e609c626-c0c6-4b01-9640-e63a163c6c96\") " pod="openshift-dns/node-resolver-nsbpk" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424887 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-os-release\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424901 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-log-socket\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.579532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424917 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdx98\" (UniqueName: \"kubernetes.io/projected/4303a48a-336e-44f9-b039-bc24bf98c3d4-kube-api-access-mdx98\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.579532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424932 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsq2v\" (UniqueName: \"kubernetes.io/projected/e609c626-c0c6-4b01-9640-e63a163c6c96-kube-api-access-bsq2v\") pod \"node-resolver-nsbpk\" (UID: \"e609c626-c0c6-4b01-9640-e63a163c6c96\") " pod="openshift-dns/node-resolver-nsbpk" Jul 06 20:35:34.579532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424946 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-system-cni-dir\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.579532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424961 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-multus-cni-dir\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.579532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-multus-socket-dir-parent\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.579532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.424999 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.579532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425019 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-socket-dir\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.579532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425034 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.579532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-run-openvswitch\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.579532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/82ffacd0-c326-48d3-8e22-be96a8e06e17-ovnkube-script-lib\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.579532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425099 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-tmp\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.579532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425119 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a6c9a93-1c46-445b-a246-7251d7f007c9-cni-binary-copy\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.579532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425134 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n865\" (UniqueName: \"kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865\") pod \"network-check-target-sxdx7\" (UID: \"2fb01258-3ad2-4383-b313-3af4d4a40c54\") " pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3268a9a6-5687-440f-91a0-5916a32aa1a5-host-slash\") pod \"iptables-alerter-5jb4d\" (UID: \"3268a9a6-5687-440f-91a0-5916a32aa1a5\") " pod="openshift-network-operator/iptables-alerter-5jb4d" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425174 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/82ffacd0-c326-48d3-8e22-be96a8e06e17-env-overrides\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425194 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-sysctl-d\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425207 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-tuned\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425224 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-os-release\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425245 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-run-netns\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425267 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flmnh\" (UniqueName: \"kubernetes.io/projected/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-kube-api-access-flmnh\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.425290 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3268a9a6-5687-440f-91a0-5916a32aa1a5-iptables-alerter-script\") pod \"iptables-alerter-5jb4d\" (UID: \"3268a9a6-5687-440f-91a0-5916a32aa1a5\") " pod="openshift-network-operator/iptables-alerter-5jb4d" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.525965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-run-k8s-cni-cncf-io\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526001 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526030 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-run-systemd\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526047 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-run-k8s-cni-cncf-io\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526058 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-cni-netd\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.580233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526096 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-run\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526132 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-run-systemd\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526140 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526103 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-cni-netd\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9gmp\" (UniqueName: \"kubernetes.io/projected/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-kube-api-access-c9gmp\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526187 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526196 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-slash\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526243 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-slash\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526247 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-run\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526260 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js5mb\" (UniqueName: \"kubernetes.io/projected/82ffacd0-c326-48d3-8e22-be96a8e06e17-kube-api-access-js5mb\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526285 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-host\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a203ceeb-e878-49c1-b1e1-ff93471aa9fb-host\") pod \"node-ca-fzclm\" (UID: \"a203ceeb-e878-49c1-b1e1-ff93471aa9fb\") " pod="openshift-image-registry/node-ca-fzclm" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526336 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tq6w\" (UniqueName: \"kubernetes.io/projected/a203ceeb-e878-49c1-b1e1-ff93471aa9fb-kube-api-access-7tq6w\") pod \"node-ca-fzclm\" (UID: \"a203ceeb-e878-49c1-b1e1-ff93471aa9fb\") " pod="openshift-image-registry/node-ca-fzclm" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526359 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-cnibin\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.580819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526368 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-host\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526379 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a203ceeb-e878-49c1-b1e1-ff93471aa9fb-host\") pod \"node-ca-fzclm\" (UID: \"a203ceeb-e878-49c1-b1e1-ff93471aa9fb\") " pod="openshift-image-registry/node-ca-fzclm" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526381 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d909edb1-81aa-4c46-8b90-18691df9680c-konnectivity-ca\") pod \"konnectivity-agent-crf7d\" (UID: \"d909edb1-81aa-4c46-8b90-18691df9680c\") " pod="kube-system/konnectivity-agent-crf7d" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526421 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-var-lib-openvswitch\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526432 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-cnibin\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-run-ovn\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-var-lib-kubelet\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526475 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-var-lib-openvswitch\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwpk\" (UniqueName: \"kubernetes.io/projected/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-kube-api-access-plwpk\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-multus-conf-dir\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526531 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-var-lib-kubelet\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-run-netns\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526534 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-run-ovn\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526573 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-etc-openvswitch\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526578 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-multus-conf-dir\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.581505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526593 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-run-ovn-kubernetes\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526603 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-run-netns\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526616 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-run-ovn-kubernetes\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526631 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526643 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-etc-openvswitch\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526651 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e609c626-c0c6-4b01-9640-e63a163c6c96-hosts-file\") pod \"node-resolver-nsbpk\" (UID: \"e609c626-c0c6-4b01-9640-e63a163c6c96\") " pod="openshift-dns/node-resolver-nsbpk" Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526694 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e609c626-c0c6-4b01-9640-e63a163c6c96-hosts-file\") pod \"node-resolver-nsbpk\" (UID: \"e609c626-c0c6-4b01-9640-e63a163c6c96\") " pod="openshift-dns/node-resolver-nsbpk" Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.526757 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526788 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-var-lib-cni-bin\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526821 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-cnibin\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526855 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-cnibin\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.526851 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs podName:4d976ba8-6cb4-4882-9f06-e9c1d62cc475 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:35.026825275 +0000 UTC m=+2.112380475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs") pod "network-metrics-daemon-d26rt" (UID: "4d976ba8-6cb4-4882-9f06-e9c1d62cc475") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526877 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-var-lib-cni-bin\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526901 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/82ffacd0-c326-48d3-8e22-be96a8e06e17-ovn-node-metrics-cert\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.582151 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526926 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-sysconfig\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526947 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d909edb1-81aa-4c46-8b90-18691df9680c-konnectivity-ca\") pod \"konnectivity-agent-crf7d\" (UID: \"d909edb1-81aa-4c46-8b90-18691df9680c\") " pod="kube-system/konnectivity-agent-crf7d" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-lib-modules\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526979 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a203ceeb-e878-49c1-b1e1-ff93471aa9fb-serviceca\") pod \"node-ca-fzclm\" (UID: \"a203ceeb-e878-49c1-b1e1-ff93471aa9fb\") " pod="openshift-image-registry/node-ca-fzclm" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.526989 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-sysconfig\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527003 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-system-cni-dir\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527026 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcsbh\" (UniqueName: \"kubernetes.io/projected/3268a9a6-5687-440f-91a0-5916a32aa1a5-kube-api-access-pcsbh\") pod \"iptables-alerter-5jb4d\" (UID: \"3268a9a6-5687-440f-91a0-5916a32aa1a5\") " pod="openshift-network-operator/iptables-alerter-5jb4d" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527075 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-system-cni-dir\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527077 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-kubelet\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527108 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-kubelet\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527121 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-node-log\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527133 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-lib-modules\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527149 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-systemd\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.582708 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527182 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-node-log\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527237 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-systemd\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527243 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-run-multus-certs\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527275 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-etc-kubernetes\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527291 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch6gp\" (UniqueName: \"kubernetes.io/projected/2a6c9a93-1c46-445b-a246-7251d7f007c9-kube-api-access-ch6gp\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527323 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527360 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-run-multus-certs\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527359 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63a48280-235d-4f85-a741-ff9d9c3aaa3b-kubelet-config\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527325 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-etc-kubernetes\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527400 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63a48280-235d-4f85-a741-ff9d9c3aaa3b-dbus\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527427 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-modprobe-d\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527435 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a203ceeb-e878-49c1-b1e1-ff93471aa9fb-serviceca\") pod \"node-ca-fzclm\" (UID: \"a203ceeb-e878-49c1-b1e1-ff93471aa9fb\") " pod="openshift-image-registry/node-ca-fzclm" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527452 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-sysctl-conf\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.583395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527517 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-registration-dir\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527554 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-modprobe-d\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527562 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527562 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-registration-dir\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527513 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63a48280-235d-4f85-a741-ff9d9c3aaa3b-kubelet-config\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527565 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63a48280-235d-4f85-a741-ff9d9c3aaa3b-dbus\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-hostroot\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527614 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-sysctl-conf\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527616 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-hostroot\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527649 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a6c9a93-1c46-445b-a246-7251d7f007c9-multus-daemon-config\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527673 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-systemd-units\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527695 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/82ffacd0-c326-48d3-8e22-be96a8e06e17-ovnkube-config\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527718 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527740 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-sys\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-sys-fs\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.583978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527740 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-systemd-units\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527787 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e609c626-c0c6-4b01-9640-e63a163c6c96-tmp-dir\") pod \"node-resolver-nsbpk\" (UID: \"e609c626-c0c6-4b01-9640-e63a163c6c96\") " pod="openshift-dns/node-resolver-nsbpk" Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.527814 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527848 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-sys\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527878 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-os-release\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527830 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-os-release\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.527910 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret podName:63a48280-235d-4f85-a741-ff9d9c3aaa3b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:35.027893663 +0000 UTC m=+2.113448852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret") pod "global-pull-secret-syncer-5zpwl" (UID: "63a48280-235d-4f85-a741-ff9d9c3aaa3b") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-log-socket\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.527986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdx98\" (UniqueName: \"kubernetes.io/projected/4303a48a-336e-44f9-b039-bc24bf98c3d4-kube-api-access-mdx98\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528013 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-log-socket\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528013 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsq2v\" (UniqueName: \"kubernetes.io/projected/e609c626-c0c6-4b01-9640-e63a163c6c96-kube-api-access-bsq2v\") pod \"node-resolver-nsbpk\" (UID: \"e609c626-c0c6-4b01-9640-e63a163c6c96\") " pod="openshift-dns/node-resolver-nsbpk" Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-system-cni-dir\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528073 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-multus-cni-dir\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528087 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e609c626-c0c6-4b01-9640-e63a163c6c96-tmp-dir\") pod \"node-resolver-nsbpk\" (UID: \"e609c626-c0c6-4b01-9640-e63a163c6c96\") " pod="openshift-dns/node-resolver-nsbpk" Jul 06 20:35:34.584579 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528100 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-multus-socket-dir-parent\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528123 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-socket-dir\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528161 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-multus-cni-dir\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528206 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-multus-socket-dir-parent\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528207 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-system-cni-dir\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528240 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a6c9a93-1c46-445b-a246-7251d7f007c9-multus-daemon-config\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528248 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528269 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-sys-fs\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528280 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-run-openvswitch\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528319 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/82ffacd0-c326-48d3-8e22-be96a8e06e17-ovnkube-config\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528354 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-run-openvswitch\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528380 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/82ffacd0-c326-48d3-8e22-be96a8e06e17-ovnkube-script-lib\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528398 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-tmp\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.585092 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528419 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a6c9a93-1c46-445b-a246-7251d7f007c9-cni-binary-copy\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528484 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n865\" (UniqueName: \"kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865\") pod \"network-check-target-sxdx7\" (UID: \"2fb01258-3ad2-4383-b313-3af4d4a40c54\") " pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528510 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3268a9a6-5687-440f-91a0-5916a32aa1a5-host-slash\") pod \"iptables-alerter-5jb4d\" (UID: \"3268a9a6-5687-440f-91a0-5916a32aa1a5\") " pod="openshift-network-operator/iptables-alerter-5jb4d" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/82ffacd0-c326-48d3-8e22-be96a8e06e17-env-overrides\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528570 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-sysctl-d\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528595 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-tuned\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528639 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-os-release\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-run-netns\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528700 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flmnh\" (UniqueName: \"kubernetes.io/projected/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-kube-api-access-flmnh\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528731 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3268a9a6-5687-440f-91a0-5916a32aa1a5-iptables-alerter-script\") pod \"iptables-alerter-5jb4d\" (UID: \"3268a9a6-5687-440f-91a0-5916a32aa1a5\") " pod="openshift-network-operator/iptables-alerter-5jb4d" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528739 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-kubernetes\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528807 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528869 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-socket-dir\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.585658 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528873 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3268a9a6-5687-440f-91a0-5916a32aa1a5-host-slash\") pod \"iptables-alerter-5jb4d\" (UID: \"3268a9a6-5687-440f-91a0-5916a32aa1a5\") " pod="openshift-network-operator/iptables-alerter-5jb4d" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528959 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/82ffacd0-c326-48d3-8e22-be96a8e06e17-ovnkube-script-lib\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.528970 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a6c9a93-1c46-445b-a246-7251d7f007c9-cni-binary-copy\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529042 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-run-netns\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529065 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-sysctl-d\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529101 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-os-release\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-device-dir\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529240 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/82ffacd0-c326-48d3-8e22-be96a8e06e17-env-overrides\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529252 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529291 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-var-lib-cni-multus\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d909edb1-81aa-4c46-8b90-18691df9680c-agent-certs\") pod \"konnectivity-agent-crf7d\" (UID: \"d909edb1-81aa-4c46-8b90-18691df9680c\") " pod="kube-system/konnectivity-agent-crf7d" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529326 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-device-dir\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529337 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4303a48a-336e-44f9-b039-bc24bf98c3d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529349 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-cni-bin\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529376 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-var-lib-cni-multus\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.586162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529379 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-var-lib-kubelet\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529417 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82ffacd0-c326-48d3-8e22-be96a8e06e17-host-cni-bin\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529505 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a6c9a93-1c46-445b-a246-7251d7f007c9-host-var-lib-kubelet\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529525 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-kubernetes\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.529586 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3268a9a6-5687-440f-91a0-5916a32aa1a5-iptables-alerter-script\") pod \"iptables-alerter-5jb4d\" (UID: \"3268a9a6-5687-440f-91a0-5916a32aa1a5\") " pod="openshift-network-operator/iptables-alerter-5jb4d" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.530796 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-tmp\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.530955 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/82ffacd0-c326-48d3-8e22-be96a8e06e17-ovn-node-metrics-cert\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.531074 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-etc-tuned\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.531694 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d909edb1-81aa-4c46-8b90-18691df9680c-agent-certs\") pod \"konnectivity-agent-crf7d\" (UID: \"d909edb1-81aa-4c46-8b90-18691df9680c\") " pod="kube-system/konnectivity-agent-crf7d" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.538599 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js5mb\" (UniqueName: \"kubernetes.io/projected/82ffacd0-c326-48d3-8e22-be96a8e06e17-kube-api-access-js5mb\") pod \"ovnkube-node-crszk\" (UID: \"82ffacd0-c326-48d3-8e22-be96a8e06e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.538603 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9gmp\" (UniqueName: \"kubernetes.io/projected/d8a0e3f3-d1d1-4732-81a9-035c2babecb9-kube-api-access-c9gmp\") pod \"tuned-jwpp2\" (UID: \"d8a0e3f3-d1d1-4732-81a9-035c2babecb9\") " pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.538627 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tq6w\" (UniqueName: \"kubernetes.io/projected/a203ceeb-e878-49c1-b1e1-ff93471aa9fb-kube-api-access-7tq6w\") pod \"node-ca-fzclm\" (UID: \"a203ceeb-e878-49c1-b1e1-ff93471aa9fb\") " pod="openshift-image-registry/node-ca-fzclm" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.538947 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch6gp\" (UniqueName: \"kubernetes.io/projected/2a6c9a93-1c46-445b-a246-7251d7f007c9-kube-api-access-ch6gp\") pod \"multus-6jqmg\" (UID: \"2a6c9a93-1c46-445b-a246-7251d7f007c9\") " pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.539025 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcsbh\" (UniqueName: \"kubernetes.io/projected/3268a9a6-5687-440f-91a0-5916a32aa1a5-kube-api-access-pcsbh\") pod \"iptables-alerter-5jb4d\" (UID: \"3268a9a6-5687-440f-91a0-5916a32aa1a5\") " pod="openshift-network-operator/iptables-alerter-5jb4d" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.539116 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwpk\" (UniqueName: \"kubernetes.io/projected/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-kube-api-access-plwpk\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:34.587042 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.541971 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:35:34.587630 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.541988 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:35:34.587630 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.541998 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6n865 for pod openshift-network-diagnostics/network-check-target-sxdx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:34.587630 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.542093 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865 podName:2fb01258-3ad2-4383-b313-3af4d4a40c54 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:35.042076871 +0000 UTC m=+2.127632049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6n865" (UniqueName: "kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865") pod "network-check-target-sxdx7" (UID: "2fb01258-3ad2-4383-b313-3af4d4a40c54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:34.587630 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.542733 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdx98\" (UniqueName: \"kubernetes.io/projected/4303a48a-336e-44f9-b039-bc24bf98c3d4-kube-api-access-mdx98\") pod \"aws-ebs-csi-driver-node-zw5rg\" (UID: \"4303a48a-336e-44f9-b039-bc24bf98c3d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.587630 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.543355 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsq2v\" (UniqueName: \"kubernetes.io/projected/e609c626-c0c6-4b01-9640-e63a163c6c96-kube-api-access-bsq2v\") pod \"node-resolver-nsbpk\" (UID: \"e609c626-c0c6-4b01-9640-e63a163c6c96\") " pod="openshift-dns/node-resolver-nsbpk" Jul 06 20:35:34.587630 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.544230 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flmnh\" (UniqueName: \"kubernetes.io/projected/9d18cb24-9db4-42fe-93b8-a47b2f26a78f-kube-api-access-flmnh\") pod \"multus-additional-cni-plugins-jqtv5\" (UID: \"9d18cb24-9db4-42fe-93b8-a47b2f26a78f\") " pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:34.668407 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900a8e8020660ce714cfb40f4b668905.slice/crio-004000271c6c169254608508ee6b00cb77522c588ca9180504de897ce3d2bc3a WatchSource:0}: Error finding container 004000271c6c169254608508ee6b00cb77522c588ca9180504de897ce3d2bc3a: Status 404 returned error can't find the container with id 004000271c6c169254608508ee6b00cb77522c588ca9180504de897ce3d2bc3a Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.669136 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:34.669307 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2150a52d3ba57389650b4becb0fa9935.slice/crio-9363ba38af45d22c116788e933215686ce32c166c43152179577ba95cb9175da WatchSource:0}: Error finding container 9363ba38af45d22c116788e933215686ce32c166c43152179577ba95cb9175da: Status 404 returned error can't find the container with id 9363ba38af45d22c116788e933215686ce32c166c43152179577ba95cb9175da Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.673559 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.675194 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jqtv5" Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:34.679776 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4303a48a_336e_44f9_b039_bc24bf98c3d4.slice/crio-ddad5e7d2097527f059de9e3643c10dc691310fac7819caf3c6c8009d6f2eb8d WatchSource:0}: Error finding container ddad5e7d2097527f059de9e3643c10dc691310fac7819caf3c6c8009d6f2eb8d: Status 404 returned error can't find the container with id ddad5e7d2097527f059de9e3643c10dc691310fac7819caf3c6c8009d6f2eb8d Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.696168 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5jb4d" Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:34.705774 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3268a9a6_5687_440f_91a0_5916a32aa1a5.slice/crio-7b7505e28a04e54191b3424271a37af6ed733ac2b70de2396909fd3d43d08ed6 WatchSource:0}: Error finding container 7b7505e28a04e54191b3424271a37af6ed733ac2b70de2396909fd3d43d08ed6: Status 404 returned error can't find the container with id 7b7505e28a04e54191b3424271a37af6ed733ac2b70de2396909fd3d43d08ed6 Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.713186 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:34.725429 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ffacd0_c326_48d3_8e22_be96a8e06e17.slice/crio-63d35aa24a7fc33a7746113c51a43b6e51411836f5dfb0b9dfb50e4bd99bfe25 WatchSource:0}: Error finding container 63d35aa24a7fc33a7746113c51a43b6e51411836f5dfb0b9dfb50e4bd99bfe25: Status 404 returned error can't find the container with id 63d35aa24a7fc33a7746113c51a43b6e51411836f5dfb0b9dfb50e4bd99bfe25 Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.737583 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:34.746672 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a0e3f3_d1d1_4732_81a9_035c2babecb9.slice/crio-63f892f8deeb199697ba562cc5560a892ea82245cace937287352b069a2dfe5a WatchSource:0}: Error finding container 63f892f8deeb199697ba562cc5560a892ea82245cace937287352b069a2dfe5a: Status 404 returned error can't find the container with id 63f892f8deeb199697ba562cc5560a892ea82245cace937287352b069a2dfe5a Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.759886 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nsbpk" Jul 06 20:35:34.770117 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:34.767730 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode609c626_c0c6_4b01_9640_e63a163c6c96.slice/crio-246705015f7713d273ab747808a448044ee8b460280d13e95b83b16e2f23432e WatchSource:0}: Error finding container 246705015f7713d273ab747808a448044ee8b460280d13e95b83b16e2f23432e: Status 404 returned error can't find the container with id 246705015f7713d273ab747808a448044ee8b460280d13e95b83b16e2f23432e Jul 06 20:35:34.975876 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.775701 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fzclm" Jul 06 20:35:34.975876 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:34.783901 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda203ceeb_e878_49c1_b1e1_ff93471aa9fb.slice/crio-07ffba4a9ef0b75047888c890c6c8fb727d3bf2b89f67cd52a0ee890bc8235ac WatchSource:0}: Error finding container 07ffba4a9ef0b75047888c890c6c8fb727d3bf2b89f67cd52a0ee890bc8235ac: Status 404 returned error can't find the container with id 07ffba4a9ef0b75047888c890c6c8fb727d3bf2b89f67cd52a0ee890bc8235ac Jul 06 20:35:34.975876 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.786798 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:35:34.975876 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.797686 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6jqmg" Jul 06 20:35:34.975876 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:34.802783 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-crf7d" Jul 06 20:35:34.975876 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:34.805804 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6c9a93_1c46_445b_a246_7251d7f007c9.slice/crio-b166987d97b846c6a1a28cea448369025bacd3fb2eb1bf31ca7c6319b92338d5 WatchSource:0}: Error finding container b166987d97b846c6a1a28cea448369025bacd3fb2eb1bf31ca7c6319b92338d5: Status 404 returned error can't find the container with id b166987d97b846c6a1a28cea448369025bacd3fb2eb1bf31ca7c6319b92338d5 Jul 06 20:35:34.975876 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:35:34.811556 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd909edb1_81aa_4c46_8b90_18691df9680c.slice/crio-78d796bb9633048255263982315f6d4a538b2c5a253d1529a5253fe963ad5da8 WatchSource:0}: Error finding container 78d796bb9633048255263982315f6d4a538b2c5a253d1529a5253fe963ad5da8: Status 404 returned error can't find the container with id 78d796bb9633048255263982315f6d4a538b2c5a253d1529a5253fe963ad5da8 Jul 06 20:35:34.975876 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.813122 2582 kuberuntime_manager.go:1459] "Unhandled Error" err="container konnectivity-agent start failed in pod konnectivity-agent-crf7d_kube-system(d909edb1-81aa-4c46-8b90-18691df9680c): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 06 20:35:34.975876 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:34.813150 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ErrImagePull: \"pull QPS exceeded\"" pod="kube-system/konnectivity-agent-crf7d" podUID="d909edb1-81aa-4c46-8b90-18691df9680c" Jul 06 20:35:35.107327 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.033528 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:35.107327 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.033571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:35.107327 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:35.033692 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:35.107327 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:35.033748 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret podName:63a48280-235d-4f85-a741-ff9d9c3aaa3b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:36.033735306 +0000 UTC m=+3.119290478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret") pod "global-pull-secret-syncer-5zpwl" (UID: "63a48280-235d-4f85-a741-ff9d9c3aaa3b") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:35.107327 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:35.033691 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:35.107327 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:35.033829 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs podName:4d976ba8-6cb4-4882-9f06-e9c1d62cc475 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:36.03381442 +0000 UTC m=+3.119369599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs") pod "network-metrics-daemon-d26rt" (UID: "4d976ba8-6cb4-4882-9f06-e9c1d62cc475") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:35.191978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.134798 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n865\" (UniqueName: \"kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865\") pod \"network-check-target-sxdx7\" (UID: \"2fb01258-3ad2-4383-b313-3af4d4a40c54\") " pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:35.191978 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:35.135010 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:35:35.191978 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:35.135035 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:35:35.191978 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:35.135048 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6n865 for pod openshift-network-diagnostics/network-check-target-sxdx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:35.191978 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:35.135109 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865 podName:2fb01258-3ad2-4383-b313-3af4d4a40c54 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:36.135090383 +0000 UTC m=+3.220645574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6n865" (UniqueName: "kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865") pod "network-check-target-sxdx7" (UID: "2fb01258-3ad2-4383-b313-3af4d4a40c54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:35.427275 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.377316 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:30:34 +0000 UTC" deadline="2027-12-11 16:35:04.12766913 +0000 UTC" Jul 06 20:35:35.427275 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.377381 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12547h59m28.750292519s" Jul 06 20:35:35.500089 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.482583 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:35:35.535091 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.533871 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:35.535091 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:35.533992 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:35.535091 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.534336 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:35.535091 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:35.534423 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:35.593110 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.587176 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6jqmg" event={"ID":"2a6c9a93-1c46-445b-a246-7251d7f007c9","Type":"ContainerStarted","Data":"b166987d97b846c6a1a28cea448369025bacd3fb2eb1bf31ca7c6319b92338d5"} Jul 06 20:35:35.597412 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.596919 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" event={"ID":"d8a0e3f3-d1d1-4732-81a9-035c2babecb9","Type":"ContainerStarted","Data":"63f892f8deeb199697ba562cc5560a892ea82245cace937287352b069a2dfe5a"} Jul 06 20:35:35.604006 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.603810 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" event={"ID":"82ffacd0-c326-48d3-8e22-be96a8e06e17","Type":"ContainerStarted","Data":"63d35aa24a7fc33a7746113c51a43b6e51411836f5dfb0b9dfb50e4bd99bfe25"} Jul 06 20:35:35.613229 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.613085 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" event={"ID":"4303a48a-336e-44f9-b039-bc24bf98c3d4","Type":"ContainerStarted","Data":"ddad5e7d2097527f059de9e3643c10dc691310fac7819caf3c6c8009d6f2eb8d"} Jul 06 20:35:35.618079 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.617326 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-crf7d" event={"ID":"d909edb1-81aa-4c46-8b90-18691df9680c","Type":"ContainerStarted","Data":"78d796bb9633048255263982315f6d4a538b2c5a253d1529a5253fe963ad5da8"} Jul 06 20:35:35.623658 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:35.623588 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2e954d9267f62174ff45a44976fa7a47e58d87fef14ffbb9fc0f8184e722280\\\": ErrImagePull: pull QPS exceeded\"" pod="kube-system/konnectivity-agent-crf7d" podUID="d909edb1-81aa-4c46-8b90-18691df9680c" Jul 06 20:35:35.627204 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.627171 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fzclm" event={"ID":"a203ceeb-e878-49c1-b1e1-ff93471aa9fb","Type":"ContainerStarted","Data":"07ffba4a9ef0b75047888c890c6c8fb727d3bf2b89f67cd52a0ee890bc8235ac"} Jul 06 20:35:35.630346 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.630318 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nsbpk" event={"ID":"e609c626-c0c6-4b01-9640-e63a163c6c96","Type":"ContainerStarted","Data":"246705015f7713d273ab747808a448044ee8b460280d13e95b83b16e2f23432e"} Jul 06 20:35:35.643180 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.643148 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5jb4d" event={"ID":"3268a9a6-5687-440f-91a0-5916a32aa1a5","Type":"ContainerStarted","Data":"7b7505e28a04e54191b3424271a37af6ed733ac2b70de2396909fd3d43d08ed6"} Jul 06 20:35:35.655865 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.655582 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqtv5" event={"ID":"9d18cb24-9db4-42fe-93b8-a47b2f26a78f","Type":"ContainerStarted","Data":"763cf8d4eecf258ce5aa0f743ff3bc1ebff29999f71b68be848085dd6a1d7725"} Jul 06 20:35:35.658819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.658739 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal" event={"ID":"900a8e8020660ce714cfb40f4b668905","Type":"ContainerStarted","Data":"004000271c6c169254608508ee6b00cb77522c588ca9180504de897ce3d2bc3a"} Jul 06 20:35:35.661977 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:35.661912 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-142.ec2.internal" event={"ID":"2150a52d3ba57389650b4becb0fa9935","Type":"ContainerStarted","Data":"9363ba38af45d22c116788e933215686ce32c166c43152179577ba95cb9175da"} Jul 06 20:35:36.060858 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:36.060030 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:36.060858 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:36.060114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:36.060858 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:36.060260 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:36.060858 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:36.060322 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs podName:4d976ba8-6cb4-4882-9f06-e9c1d62cc475 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:38.060301208 +0000 UTC m=+5.145856387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs") pod "network-metrics-daemon-d26rt" (UID: "4d976ba8-6cb4-4882-9f06-e9c1d62cc475") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:36.060858 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:36.060741 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:36.060858 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:36.060790 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret podName:63a48280-235d-4f85-a741-ff9d9c3aaa3b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:38.060775867 +0000 UTC m=+5.146331058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret") pod "global-pull-secret-syncer-5zpwl" (UID: "63a48280-235d-4f85-a741-ff9d9c3aaa3b") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:36.070884 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:36.070859 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:35:36.161368 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:36.160687 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n865\" (UniqueName: \"kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865\") pod \"network-check-target-sxdx7\" (UID: \"2fb01258-3ad2-4383-b313-3af4d4a40c54\") " pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:36.161368 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:36.160881 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:35:36.161368 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:36.160906 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:35:36.161368 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:36.160918 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6n865 for pod openshift-network-diagnostics/network-check-target-sxdx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:36.161368 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:36.160977 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865 podName:2fb01258-3ad2-4383-b313-3af4d4a40c54 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:38.160956455 +0000 UTC m=+5.246511642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6n865" (UniqueName: "kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865") pod "network-check-target-sxdx7" (UID: "2fb01258-3ad2-4383-b313-3af4d4a40c54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:36.378075 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:36.377931 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:30:34 +0000 UTC" deadline="2028-03-09 23:46:54.92976745 +0000 UTC" Jul 06 20:35:36.378075 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:36.377974 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14691h11m18.551798336s" Jul 06 20:35:36.532697 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:36.532176 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:36.532697 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:36.532314 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:35:37.535263 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:37.535227 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:37.535741 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:37.535418 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:37.544178 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:37.543960 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:37.544178 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:37.544120 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:38.076675 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:38.076640 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:38.076865 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:38.076704 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:38.076926 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:38.076860 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:38.076974 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:38.076935 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs podName:4d976ba8-6cb4-4882-9f06-e9c1d62cc475 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:42.076911705 +0000 UTC m=+9.162466901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs") pod "network-metrics-daemon-d26rt" (UID: "4d976ba8-6cb4-4882-9f06-e9c1d62cc475") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:38.076974 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:38.076869 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:38.077094 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:38.077007 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret podName:63a48280-235d-4f85-a741-ff9d9c3aaa3b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:42.076990012 +0000 UTC m=+9.162545188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret") pod "global-pull-secret-syncer-5zpwl" (UID: "63a48280-235d-4f85-a741-ff9d9c3aaa3b") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:38.177904 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:38.177866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n865\" (UniqueName: \"kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865\") pod \"network-check-target-sxdx7\" (UID: \"2fb01258-3ad2-4383-b313-3af4d4a40c54\") " pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:38.178066 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:38.178047 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:35:38.178146 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:38.178074 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:35:38.178146 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:38.178087 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6n865 for pod openshift-network-diagnostics/network-check-target-sxdx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:38.178255 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:38.178162 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865 podName:2fb01258-3ad2-4383-b313-3af4d4a40c54 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:42.178142317 +0000 UTC m=+9.263697501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6n865" (UniqueName: "kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865") pod "network-check-target-sxdx7" (UID: "2fb01258-3ad2-4383-b313-3af4d4a40c54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:38.532037 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:38.531951 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:38.532208 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:38.532098 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:35:39.532686 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:39.532650 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:39.533187 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:39.532827 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:39.533560 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:39.533369 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:39.533560 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:39.533490 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:40.531454 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:40.531418 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:40.531654 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:40.531551 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:35:41.532195 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:41.532108 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:41.532195 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:41.532128 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:41.532698 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:41.532250 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:41.532698 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:41.532403 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:42.109447 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:42.109410 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:42.109617 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:42.109473 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:42.109688 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:42.109612 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:42.109735 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:42.109707 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret podName:63a48280-235d-4f85-a741-ff9d9c3aaa3b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:50.109689691 +0000 UTC m=+17.195244868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret") pod "global-pull-secret-syncer-5zpwl" (UID: "63a48280-235d-4f85-a741-ff9d9c3aaa3b") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:42.110138 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:42.110118 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:42.110227 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:42.110174 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs podName:4d976ba8-6cb4-4882-9f06-e9c1d62cc475 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:50.110159241 +0000 UTC m=+17.195714415 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs") pod "network-metrics-daemon-d26rt" (UID: "4d976ba8-6cb4-4882-9f06-e9c1d62cc475") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:42.210573 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:42.210538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n865\" (UniqueName: \"kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865\") pod \"network-check-target-sxdx7\" (UID: \"2fb01258-3ad2-4383-b313-3af4d4a40c54\") " pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:42.210728 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:42.210712 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:35:42.210784 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:42.210739 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:35:42.210784 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:42.210750 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6n865 for pod openshift-network-diagnostics/network-check-target-sxdx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:42.210858 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:42.210825 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865 podName:2fb01258-3ad2-4383-b313-3af4d4a40c54 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:50.210805876 +0000 UTC m=+17.296361067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6n865" (UniqueName: "kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865") pod "network-check-target-sxdx7" (UID: "2fb01258-3ad2-4383-b313-3af4d4a40c54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:42.531824 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:42.531730 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:42.531998 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:42.531887 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:35:43.532928 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:43.532893 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:43.533398 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:43.533067 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:43.533570 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:43.533546 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:43.533709 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:43.533661 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:44.532325 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:44.532288 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:44.532484 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:44.532422 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:35:45.531496 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:45.531460 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:45.531901 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:45.531579 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:45.531901 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:45.531613 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:45.531901 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:45.531708 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:46.532540 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:46.532448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:46.532990 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:46.532595 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:35:47.531892 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:47.531854 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:47.532042 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:47.531981 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:47.532100 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:47.532034 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:47.532212 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:47.532173 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:48.532168 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:48.532128 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:48.532632 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:48.532272 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:35:49.532275 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:49.532239 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:49.532726 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:49.532239 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:49.532726 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:49.532381 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:49.532726 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:49.532427 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:50.170115 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:50.170069 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:50.170295 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:50.170123 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:50.170295 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:50.170237 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:50.170295 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:50.170293 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret podName:63a48280-235d-4f85-a741-ff9d9c3aaa3b nodeName:}" failed. No retries permitted until 2026-07-06 20:36:06.170275002 +0000 UTC m=+33.255830173 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret") pod "global-pull-secret-syncer-5zpwl" (UID: "63a48280-235d-4f85-a741-ff9d9c3aaa3b") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:35:50.170456 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:50.170234 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:50.170456 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:50.170406 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs podName:4d976ba8-6cb4-4882-9f06-e9c1d62cc475 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:06.170384086 +0000 UTC m=+33.255939260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs") pod "network-metrics-daemon-d26rt" (UID: "4d976ba8-6cb4-4882-9f06-e9c1d62cc475") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:35:50.270629 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:50.270598 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n865\" (UniqueName: \"kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865\") pod \"network-check-target-sxdx7\" (UID: \"2fb01258-3ad2-4383-b313-3af4d4a40c54\") " pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:50.270824 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:50.270790 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:35:50.270824 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:50.270815 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:35:50.270920 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:50.270827 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6n865 for pod openshift-network-diagnostics/network-check-target-sxdx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:50.270920 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:50.270890 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865 podName:2fb01258-3ad2-4383-b313-3af4d4a40c54 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:06.270870098 +0000 UTC m=+33.356425289 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6n865" (UniqueName: "kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865") pod "network-check-target-sxdx7" (UID: "2fb01258-3ad2-4383-b313-3af4d4a40c54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:35:50.531650 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:50.531563 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:50.531820 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:50.531688 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:35:51.531686 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:51.531556 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:51.531686 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:51.531678 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:51.532205 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:51.531740 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:51.532205 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:51.531863 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:52.531569 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:52.531536 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:52.531715 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:52.531658 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:35:53.534547 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:53.534344 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:53.535002 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:53.534409 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:53.535002 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:53.534636 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:53.535002 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:53.534701 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:54.531973 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:54.531953 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:54.532078 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:54.532050 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:35:54.710455 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:54.710356 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-142.ec2.internal" event={"ID":"2150a52d3ba57389650b4becb0fa9935","Type":"ContainerStarted","Data":"df4ed78b53da753b38b87c2d4457e9efd5282be3b8fc6fc66d25e8b961ea8854"} Jul 06 20:35:54.712267 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:54.712220 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" event={"ID":"d8a0e3f3-d1d1-4732-81a9-035c2babecb9","Type":"ContainerStarted","Data":"d662f319758353ef4f5e5c2a84388fd33963d11ffb6f06b79c8ebd2e4559382e"} Jul 06 20:35:54.714040 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:54.714015 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" event={"ID":"82ffacd0-c326-48d3-8e22-be96a8e06e17","Type":"ContainerStarted","Data":"b998b5736f28aa60f354d8722cc7bc6cd9f62a61e8546a058282a99029c16c8c"} Jul 06 20:35:54.728327 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:54.727814 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-142.ec2.internal" podStartSLOduration=20.72779948 podStartE2EDuration="20.72779948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:54.727369675 +0000 UTC m=+21.812924869" watchObservedRunningTime="2026-07-06 20:35:54.72779948 +0000 UTC m=+21.813354676" Jul 06 20:35:54.748559 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:54.747828 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jwpp2" podStartSLOduration=1.129033017 podStartE2EDuration="20.747810178s" totalImagesPullingTime="19.618777161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:54.747606972 +0000 UTC m=+21.833162166" watchObservedRunningTime="2026-07-06 20:35:54.747810178 +0000 UTC m=+21.833365372" Jul 06 20:35:55.532146 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.531965 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:55.532292 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.531986 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:55.532292 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:55.532234 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:55.532360 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:55.532313 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:55.717415 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.717379 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nsbpk" event={"ID":"e609c626-c0c6-4b01-9640-e63a163c6c96","Type":"ContainerStarted","Data":"b514dd39125918cf26b605e77475bebcd7f75012834ac82fac0752672b72a998"} Jul 06 20:35:55.718876 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.718847 2582 generic.go:356] "Generic (PLEG): container finished" podID="9d18cb24-9db4-42fe-93b8-a47b2f26a78f" containerID="31dbe9026df8fda675d8b33615b5626cefa9075d162522c4ba93e6fb358c3ce1" exitCode=0 Jul 06 20:35:55.718994 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.718943 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqtv5" event={"ID":"9d18cb24-9db4-42fe-93b8-a47b2f26a78f","Type":"ContainerDied","Data":"31dbe9026df8fda675d8b33615b5626cefa9075d162522c4ba93e6fb358c3ce1"} Jul 06 20:35:55.723185 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.723157 2582 generic.go:356] "Generic (PLEG): container finished" podID="900a8e8020660ce714cfb40f4b668905" containerID="3a61b7be494576611c3efb871043066d474e03d7f432ef550055282b68b15c36" exitCode=0 Jul 06 20:35:55.723264 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.723233 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal" event={"ID":"900a8e8020660ce714cfb40f4b668905","Type":"ContainerDied","Data":"3a61b7be494576611c3efb871043066d474e03d7f432ef550055282b68b15c36"} Jul 06 20:35:55.724686 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.724667 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6jqmg" event={"ID":"2a6c9a93-1c46-445b-a246-7251d7f007c9","Type":"ContainerStarted","Data":"1e3a86a38f0fc4ef4baa04ad68d10b713abcb8ac4738a0208d95fbde0ba759fd"} Jul 06 20:35:55.727152 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.727137 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 20:35:55.727392 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.727377 2582 generic.go:356] "Generic (PLEG): container finished" podID="82ffacd0-c326-48d3-8e22-be96a8e06e17" containerID="0986c622ccdf97c7d6c9fd06c73f08bd9274654882fedffc7a76edac7e1145be" exitCode=1 Jul 06 20:35:55.727458 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.727430 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" event={"ID":"82ffacd0-c326-48d3-8e22-be96a8e06e17","Type":"ContainerStarted","Data":"b1821da69c799d987d788334a84f18a5287ed95c9377adfa8808fdc0ee091a2f"} Jul 06 20:35:55.727495 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.727469 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" event={"ID":"82ffacd0-c326-48d3-8e22-be96a8e06e17","Type":"ContainerStarted","Data":"356b62acdf1e1f9b63ecc94e94af6793a94ff9d813925086a5ee54f9c0eeb393"} Jul 06 20:35:55.727531 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.727511 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" event={"ID":"82ffacd0-c326-48d3-8e22-be96a8e06e17","Type":"ContainerStarted","Data":"72cfb656d5c3c5d17cc3528e5aa86e119d32dee9131a8ed2f60791ca4b0505a8"} Jul 06 20:35:55.727570 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.727542 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" event={"ID":"82ffacd0-c326-48d3-8e22-be96a8e06e17","Type":"ContainerStarted","Data":"69f19a34e10dfa34cb88b9a7fe99800fa6d3d956247923ec903b62a4ad9cbe67"} Jul 06 20:35:55.727614 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.727574 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" event={"ID":"82ffacd0-c326-48d3-8e22-be96a8e06e17","Type":"ContainerDied","Data":"0986c622ccdf97c7d6c9fd06c73f08bd9274654882fedffc7a76edac7e1145be"} Jul 06 20:35:55.728623 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.728605 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" event={"ID":"4303a48a-336e-44f9-b039-bc24bf98c3d4","Type":"ContainerStarted","Data":"e142bc6a7f20de5c728e23fc027b0043fb1fb5f272b88dcb2aee681fe4a10dee"} Jul 06 20:35:55.729850 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.729831 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fzclm" event={"ID":"a203ceeb-e878-49c1-b1e1-ff93471aa9fb","Type":"ContainerStarted","Data":"010ecf9cbf85336a8627715ed72ef7849b2ac9ed1649e03946750ac24a9c28d4"} Jul 06 20:35:55.735574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.735541 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nsbpk" podStartSLOduration=2.136794142 podStartE2EDuration="21.735531034s" totalImagesPullingTime="19.598736892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:55.735127744 +0000 UTC m=+22.820682937" watchObservedRunningTime="2026-07-06 20:35:55.735531034 +0000 UTC m=+22.821086227" Jul 06 20:35:55.750619 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.750580 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fzclm" podStartSLOduration=2.18831425 podStartE2EDuration="21.750565458s" totalImagesPullingTime="19.562251208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:55.750286854 +0000 UTC m=+22.835842047" watchObservedRunningTime="2026-07-06 20:35:55.750565458 +0000 UTC m=+22.836120650" Jul 06 20:35:55.771294 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:55.771258 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6jqmg" podStartSLOduration=1.833777557 podStartE2EDuration="21.771245801s" totalImagesPullingTime="19.937468244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:55.771099429 +0000 UTC m=+22.856654621" watchObservedRunningTime="2026-07-06 20:35:55.771245801 +0000 UTC m=+22.856800992" Jul 06 20:35:56.462823 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:56.462618 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 20:35:56.531873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:56.531853 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:56.531977 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:56.531940 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:35:56.733828 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:56.733794 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5jb4d" event={"ID":"3268a9a6-5687-440f-91a0-5916a32aa1a5","Type":"ContainerStarted","Data":"9393ee19428c47a5fc77f9f2b7aa975ba52b851e22b4e314b1fd3b9132d26ffe"} Jul 06 20:35:56.735829 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:56.735799 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal" event={"ID":"900a8e8020660ce714cfb40f4b668905","Type":"ContainerStarted","Data":"77d7b0c340412792686b818781a4348b49652f9edd4cc8a4d28fb1ee1bd745c3"} Jul 06 20:35:56.737859 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:56.737832 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" event={"ID":"4303a48a-336e-44f9-b039-bc24bf98c3d4","Type":"ContainerStarted","Data":"159e065d07bf7fb63eeba56ba35cb59c9d3ca238621856208a00cd3d731ec95d"} Jul 06 20:35:56.739460 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:56.739436 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-crf7d" event={"ID":"d909edb1-81aa-4c46-8b90-18691df9680c","Type":"ContainerStarted","Data":"e70372babd58e60d64371f5587b8344973ecfa4b2aaf4640f8f6fc06b7ede419"} Jul 06 20:35:56.753011 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:56.752972 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5jb4d" podStartSLOduration=3.112621399 podStartE2EDuration="22.752956664s" totalImagesPullingTime="19.640335265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:56.752653884 +0000 UTC m=+23.838209089" watchObservedRunningTime="2026-07-06 20:35:56.752956664 +0000 UTC m=+23.838511856" Jul 06 20:35:56.770913 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:56.770874 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-142.ec2.internal" podStartSLOduration=22.770861432 podStartE2EDuration="22.770861432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:56.770153701 +0000 UTC m=+23.855708893" watchObservedRunningTime="2026-07-06 20:35:56.770861432 +0000 UTC m=+23.856416624" Jul 06 20:35:56.833469 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:56.833448 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-crf7d" Jul 06 20:35:57.401276 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:57.401183 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T20:35:56.462796134Z","UUID":"a82f6ee6-634d-40de-9d70-2a4bc015c028","Handler":null,"Name":"","Endpoint":""} Jul 06 20:35:57.402598 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:57.402564 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 20:35:57.402706 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:57.402605 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 20:35:57.532951 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:57.532426 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:57.532951 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:57.532553 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:57.532951 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:57.532609 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:57.532951 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:57.532709 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:57.744629 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:57.744561 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 20:35:57.745037 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:57.745001 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" event={"ID":"82ffacd0-c326-48d3-8e22-be96a8e06e17","Type":"ContainerStarted","Data":"64d681cc54545107c23a655dc8b9e75c62240ac289126d5af443a9675c1ca991"} Jul 06 20:35:58.531992 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:58.531963 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:35:58.532241 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:58.532082 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:35:58.749464 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:58.749431 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" event={"ID":"4303a48a-336e-44f9-b039-bc24bf98c3d4","Type":"ContainerStarted","Data":"e8b8914b802a3f47225b3dae517310295bc7742a31fa0daa89f3d1de94093c6c"} Jul 06 20:35:58.788308 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:58.788223 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-crf7d" podStartSLOduration=3.834425867 podStartE2EDuration="24.788210107s" totalImagesPullingTime="20.95378424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-06 20:35:56.787189855 +0000 UTC m=+23.872745059" watchObservedRunningTime="2026-07-06 20:35:58.788210107 +0000 UTC m=+25.873765299" Jul 06 20:35:58.788453 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:58.788331 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zw5rg" podStartSLOduration=2.783673686 podStartE2EDuration="24.788323234s" totalImagesPullingTime="22.004649548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:58.787880456 +0000 UTC m=+25.873435696" watchObservedRunningTime="2026-07-06 20:35:58.788323234 +0000 UTC m=+25.873878429" Jul 06 20:35:59.531863 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:59.531837 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:35:59.531863 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:59.531848 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:35:59.532103 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:59.531955 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:35:59.532158 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:35:59.532095 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:35:59.549672 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:59.549646 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-crf7d" Jul 06 20:35:59.550395 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:59.550375 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-crf7d" Jul 06 20:35:59.752528 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:35:59.752500 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-crf7d" Jul 06 20:36:00.531958 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:00.531776 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:36:00.532093 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:00.532006 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:36:00.755110 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:00.755079 2582 generic.go:356] "Generic (PLEG): container finished" podID="9d18cb24-9db4-42fe-93b8-a47b2f26a78f" containerID="13a51b714ec49d8d2525db4d7c1cd62d3e8c5f2ca28ebf94b2bf9c0356ac4d46" exitCode=0 Jul 06 20:36:00.755829 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:00.755169 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqtv5" event={"ID":"9d18cb24-9db4-42fe-93b8-a47b2f26a78f","Type":"ContainerDied","Data":"13a51b714ec49d8d2525db4d7c1cd62d3e8c5f2ca28ebf94b2bf9c0356ac4d46"} Jul 06 20:36:00.758574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:00.758558 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 20:36:00.758929 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:00.758905 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" event={"ID":"82ffacd0-c326-48d3-8e22-be96a8e06e17","Type":"ContainerStarted","Data":"8f119e5671cb935249fa248c95118962be8486426c9a7c74a9f5a1f7a56ea70c"} Jul 06 20:36:00.759326 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:00.759310 2582 scope.go:117] "RemoveContainer" containerID="0986c622ccdf97c7d6c9fd06c73f08bd9274654882fedffc7a76edac7e1145be" Jul 06 20:36:01.532476 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:01.532403 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:36:01.532622 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:01.532508 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:36:01.532622 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:01.532558 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:36:01.532730 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:01.532654 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:36:01.764091 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:01.764067 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 20:36:01.764666 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:01.764404 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" event={"ID":"82ffacd0-c326-48d3-8e22-be96a8e06e17","Type":"ContainerStarted","Data":"d5ef61f723a9b27b6c6ff7c9d9e1614854a9a174f945365390a54b5e32c000c1"} Jul 06 20:36:01.764728 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:01.764674 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:36:01.764810 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:01.764746 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:36:01.764810 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:01.764808 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:36:01.779193 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:01.779173 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:36:01.779351 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:01.779335 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:36:01.859380 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:01.859343 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" podStartSLOduration=8.179465324 podStartE2EDuration="27.85933059s" totalImagesPullingTime="19.679865266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:01.814466932 +0000 UTC m=+28.900022124" watchObservedRunningTime="2026-07-06 20:36:01.85933059 +0000 UTC m=+28.944885782" Jul 06 20:36:02.532344 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:02.532212 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:36:02.532467 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:02.532413 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:36:02.707659 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:02.707632 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5zpwl"] Jul 06 20:36:02.707810 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:02.707744 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:36:02.707888 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:02.707865 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:36:02.710160 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:02.710140 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d26rt"] Jul 06 20:36:02.710265 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:02.710251 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:36:02.710363 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:02.710348 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:36:02.713253 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:02.713231 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sxdx7"] Jul 06 20:36:02.768655 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:02.768580 2582 generic.go:356] "Generic (PLEG): container finished" podID="9d18cb24-9db4-42fe-93b8-a47b2f26a78f" containerID="5123968293cdc684a3c8f4187b006bae077cd0bf776c43f223530cef37c0ddda" exitCode=0 Jul 06 20:36:02.768973 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:02.768657 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:36:02.768973 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:02.768666 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqtv5" event={"ID":"9d18cb24-9db4-42fe-93b8-a47b2f26a78f","Type":"ContainerDied","Data":"5123968293cdc684a3c8f4187b006bae077cd0bf776c43f223530cef37c0ddda"} Jul 06 20:36:02.768973 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:02.768744 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:36:04.531470 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:04.531403 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:36:04.531940 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:04.531403 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:36:04.531940 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:04.531509 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:36:04.531940 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:04.531403 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:36:04.531940 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:04.531577 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:36:04.531940 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:04.531623 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:36:04.774906 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:04.774878 2582 generic.go:356] "Generic (PLEG): container finished" podID="9d18cb24-9db4-42fe-93b8-a47b2f26a78f" containerID="54e4c2e58018142c1d2038c0cce9550854218b7f005b5c77ca945553544872b9" exitCode=0 Jul 06 20:36:04.775040 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:04.774946 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqtv5" event={"ID":"9d18cb24-9db4-42fe-93b8-a47b2f26a78f","Type":"ContainerDied","Data":"54e4c2e58018142c1d2038c0cce9550854218b7f005b5c77ca945553544872b9"} Jul 06 20:36:06.191534 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:06.191500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:36:06.192106 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:06.191546 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:36:06.192106 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:06.191647 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:36:06.192106 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:06.191653 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:36:06.192106 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:06.191702 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret podName:63a48280-235d-4f85-a741-ff9d9c3aaa3b nodeName:}" failed. No retries permitted until 2026-07-06 20:36:38.191687367 +0000 UTC m=+65.277242555 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret") pod "global-pull-secret-syncer-5zpwl" (UID: "63a48280-235d-4f85-a741-ff9d9c3aaa3b") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:36:06.192106 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:06.191720 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs podName:4d976ba8-6cb4-4882-9f06-e9c1d62cc475 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:38.191710248 +0000 UTC m=+65.277265419 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs") pod "network-metrics-daemon-d26rt" (UID: "4d976ba8-6cb4-4882-9f06-e9c1d62cc475") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:36:06.292163 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:06.292139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n865\" (UniqueName: \"kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865\") pod \"network-check-target-sxdx7\" (UID: \"2fb01258-3ad2-4383-b313-3af4d4a40c54\") " pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:36:06.292392 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:06.292283 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:36:06.292392 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:06.292303 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:36:06.292392 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:06.292312 2582 projected.go:196] Error preparing data for projected volume kube-api-access-6n865 for pod openshift-network-diagnostics/network-check-target-sxdx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:36:06.292392 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:06.292365 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865 podName:2fb01258-3ad2-4383-b313-3af4d4a40c54 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:38.292352874 +0000 UTC m=+65.377908045 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6n865" (UniqueName: "kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865") pod "network-check-target-sxdx7" (UID: "2fb01258-3ad2-4383-b313-3af4d4a40c54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:36:06.532404 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:06.532329 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:36:06.532540 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:06.532445 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:36:06.532540 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:06.532454 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:36:06.532540 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:06.532473 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:36:06.532697 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:06.532566 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:36:06.532697 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:06.532683 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:36:08.531820 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.531732 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:36:08.531820 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.531751 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:36:08.531820 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.531798 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:36:08.532232 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:08.531893 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxdx7" podUID="2fb01258-3ad2-4383-b313-3af4d4a40c54" Jul 06 20:36:08.532232 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:08.532008 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zpwl" podUID="63a48280-235d-4f85-a741-ff9d9c3aaa3b" Jul 06 20:36:08.532232 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:08.532094 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d26rt" podUID="4d976ba8-6cb4-4882-9f06-e9c1d62cc475" Jul 06 20:36:08.660569 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.660350 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-142.ec2.internal" event="NodeReady" Jul 06 20:36:08.660732 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.660717 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 20:36:08.745794 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.745250 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t8qhq"] Jul 06 20:36:08.747919 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.747898 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:08.748004 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.747966 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b7qtk"] Jul 06 20:36:08.749417 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.749403 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b7qtk" Jul 06 20:36:08.762138 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.762120 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 20:36:08.762237 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.762153 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 20:36:08.762283 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.762267 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 20:36:08.763175 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.763161 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 20:36:08.763216 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.763193 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67jtg\"" Jul 06 20:36:08.763386 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.763375 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 20:36:08.770162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.770146 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mszj5\"" Jul 06 20:36:08.779493 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.779470 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t8qhq"] Jul 06 20:36:08.787191 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.787148 2582 generic.go:356] "Generic (PLEG): container finished" podID="9d18cb24-9db4-42fe-93b8-a47b2f26a78f" containerID="986a50dbf52ebf72b27b2ce1f5f70f427013f0f3390a896caa218d4f95051bc6" exitCode=0 Jul 06 20:36:08.787259 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.787203 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqtv5" event={"ID":"9d18cb24-9db4-42fe-93b8-a47b2f26a78f","Type":"ContainerDied","Data":"986a50dbf52ebf72b27b2ce1f5f70f427013f0f3390a896caa218d4f95051bc6"} Jul 06 20:36:08.808346 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.808325 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b7qtk"] Jul 06 20:36:08.811725 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.811702 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-config-volume\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:08.811825 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.811735 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-tmp-dir\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:08.811825 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.811791 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/796f6480-1067-43ca-b1f9-eff19e1a8720-cert\") pod \"ingress-canary-b7qtk\" (UID: \"796f6480-1067-43ca-b1f9-eff19e1a8720\") " pod="openshift-ingress-canary/ingress-canary-b7qtk" Jul 06 20:36:08.811932 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.811867 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:08.811932 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.811900 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkxdf\" (UniqueName: \"kubernetes.io/projected/796f6480-1067-43ca-b1f9-eff19e1a8720-kube-api-access-nkxdf\") pod \"ingress-canary-b7qtk\" (UID: \"796f6480-1067-43ca-b1f9-eff19e1a8720\") " pod="openshift-ingress-canary/ingress-canary-b7qtk" Jul 06 20:36:08.812010 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.811994 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn46g\" (UniqueName: \"kubernetes.io/projected/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-kube-api-access-zn46g\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:08.912457 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.912437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn46g\" (UniqueName: \"kubernetes.io/projected/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-kube-api-access-zn46g\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:08.912536 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.912513 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-config-volume\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:08.912586 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.912536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-tmp-dir\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:08.912586 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.912553 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/796f6480-1067-43ca-b1f9-eff19e1a8720-cert\") pod \"ingress-canary-b7qtk\" (UID: \"796f6480-1067-43ca-b1f9-eff19e1a8720\") " pod="openshift-ingress-canary/ingress-canary-b7qtk" Jul 06 20:36:08.912676 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.912585 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:08.912676 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.912606 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkxdf\" (UniqueName: \"kubernetes.io/projected/796f6480-1067-43ca-b1f9-eff19e1a8720-kube-api-access-nkxdf\") pod \"ingress-canary-b7qtk\" (UID: \"796f6480-1067-43ca-b1f9-eff19e1a8720\") " pod="openshift-ingress-canary/ingress-canary-b7qtk" Jul 06 20:36:08.912788 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:08.912699 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:36:08.912788 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:08.912776 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls podName:5c07f3a3-0155-4529-a90d-d5dbdacf7b15 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:09.412739812 +0000 UTC m=+36.498294986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls") pod "dns-default-t8qhq" (UID: "5c07f3a3-0155-4529-a90d-d5dbdacf7b15") : secret "dns-default-metrics-tls" not found Jul 06 20:36:08.912955 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.912935 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-tmp-dir\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:08.913156 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.913140 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-config-volume\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:08.916422 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.916399 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/796f6480-1067-43ca-b1f9-eff19e1a8720-cert\") pod \"ingress-canary-b7qtk\" (UID: \"796f6480-1067-43ca-b1f9-eff19e1a8720\") " pod="openshift-ingress-canary/ingress-canary-b7qtk" Jul 06 20:36:08.931641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.931622 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn46g\" (UniqueName: \"kubernetes.io/projected/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-kube-api-access-zn46g\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:08.932255 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:08.932234 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkxdf\" (UniqueName: \"kubernetes.io/projected/796f6480-1067-43ca-b1f9-eff19e1a8720-kube-api-access-nkxdf\") pod \"ingress-canary-b7qtk\" (UID: \"796f6480-1067-43ca-b1f9-eff19e1a8720\") " pod="openshift-ingress-canary/ingress-canary-b7qtk" Jul 06 20:36:09.075903 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:09.075822 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b7qtk" Jul 06 20:36:09.217088 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:09.217054 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b7qtk"] Jul 06 20:36:09.221826 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:36:09.221800 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796f6480_1067_43ca_b1f9_eff19e1a8720.slice/crio-519eee4691930a6fb13d745cdc9da38380f83ea281cdc5abef8c258521298b5b WatchSource:0}: Error finding container 519eee4691930a6fb13d745cdc9da38380f83ea281cdc5abef8c258521298b5b: Status 404 returned error can't find the container with id 519eee4691930a6fb13d745cdc9da38380f83ea281cdc5abef8c258521298b5b Jul 06 20:36:09.417285 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:09.417261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:09.417417 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:09.417391 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:36:09.417467 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:09.417444 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls podName:5c07f3a3-0155-4529-a90d-d5dbdacf7b15 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:10.41742808 +0000 UTC m=+37.502983252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls") pod "dns-default-t8qhq" (UID: "5c07f3a3-0155-4529-a90d-d5dbdacf7b15") : secret "dns-default-metrics-tls" not found Jul 06 20:36:09.790806 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:09.790713 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b7qtk" event={"ID":"796f6480-1067-43ca-b1f9-eff19e1a8720","Type":"ContainerStarted","Data":"519eee4691930a6fb13d745cdc9da38380f83ea281cdc5abef8c258521298b5b"} Jul 06 20:36:09.794233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:09.794205 2582 generic.go:356] "Generic (PLEG): container finished" podID="9d18cb24-9db4-42fe-93b8-a47b2f26a78f" containerID="15c7cc167f9a7c2c6b65628e998783423a12167ff2ebf6ae7ba3842dbd81a64e" exitCode=0 Jul 06 20:36:09.794343 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:09.794257 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqtv5" event={"ID":"9d18cb24-9db4-42fe-93b8-a47b2f26a78f","Type":"ContainerDied","Data":"15c7cc167f9a7c2c6b65628e998783423a12167ff2ebf6ae7ba3842dbd81a64e"} Jul 06 20:36:10.426974 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:10.426714 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:10.427177 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:10.426862 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:36:10.427177 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:10.427090 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls podName:5c07f3a3-0155-4529-a90d-d5dbdacf7b15 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:12.427073982 +0000 UTC m=+39.512629166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls") pod "dns-default-t8qhq" (UID: "5c07f3a3-0155-4529-a90d-d5dbdacf7b15") : secret "dns-default-metrics-tls" not found Jul 06 20:36:10.532173 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:10.532139 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:36:10.532334 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:10.532249 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:36:10.532397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:10.532339 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:36:10.539876 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:10.539854 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 20:36:10.541009 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:10.540989 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 20:36:10.541105 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:10.540994 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 20:36:10.541248 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:10.541209 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c9msn\"" Jul 06 20:36:10.541333 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:10.541253 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sxc6n\"" Jul 06 20:36:10.541333 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:10.541285 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 20:36:10.801441 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:10.801363 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqtv5" event={"ID":"9d18cb24-9db4-42fe-93b8-a47b2f26a78f","Type":"ContainerStarted","Data":"6064d307bfb3cdca8f15fdf2f0a4add39f61db567a45b9d5be4b3ed4556ddb35"} Jul 06 20:36:10.831240 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:10.831193 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jqtv5" podStartSLOduration=6.747542057 podStartE2EDuration="36.83117363s" totalImagesPullingTime="30.083631573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:10.829248027 +0000 UTC m=+37.914803221" watchObservedRunningTime="2026-07-06 20:36:10.83117363 +0000 UTC m=+37.916728825" Jul 06 20:36:11.805007 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:11.804976 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b7qtk" event={"ID":"796f6480-1067-43ca-b1f9-eff19e1a8720","Type":"ContainerStarted","Data":"daaab7d12e641f0e7cce5f25cd13ae49d59288315ae356f01ce4b856f8777ac0"} Jul 06 20:36:11.831990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:11.831943 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b7qtk" podStartSLOduration=1.4164103319999999 podStartE2EDuration="3.831924612s" totalImagesPullingTime="2.41551428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:11.830335232 +0000 UTC m=+38.915890426" watchObservedRunningTime="2026-07-06 20:36:11.831924612 +0000 UTC m=+38.917479804" Jul 06 20:36:12.443521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:12.443494 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:12.443639 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:12.443600 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:36:12.443672 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:12.443649 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls podName:5c07f3a3-0155-4529-a90d-d5dbdacf7b15 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:16.443636506 +0000 UTC m=+43.529191677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls") pod "dns-default-t8qhq" (UID: "5c07f3a3-0155-4529-a90d-d5dbdacf7b15") : secret "dns-default-metrics-tls" not found Jul 06 20:36:16.469986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:16.469951 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:16.470363 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:16.470091 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:36:16.470363 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:16.470145 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls podName:5c07f3a3-0155-4529-a90d-d5dbdacf7b15 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:24.470132792 +0000 UTC m=+51.555687963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls") pod "dns-default-t8qhq" (UID: "5c07f3a3-0155-4529-a90d-d5dbdacf7b15") : secret "dns-default-metrics-tls" not found Jul 06 20:36:24.144720 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.144692 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2"] Jul 06 20:36:24.160656 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.160632 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:24.161318 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.161293 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2"] Jul 06 20:36:24.165250 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.165228 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 06 20:36:24.165609 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.165593 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 20:36:24.166986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.166952 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qcxw2\"" Jul 06 20:36:24.166986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.166965 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 20:36:24.167146 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.166991 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 06 20:36:24.319355 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.319333 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln75p\" (UniqueName: \"kubernetes.io/projected/54b52e62-b5cd-4d43-adcf-32649b70e2d6-kube-api-access-ln75p\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:24.319437 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.319389 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/54b52e62-b5cd-4d43-adcf-32649b70e2d6-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:24.319437 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.319429 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:24.345024 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.345000 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-666c749768-mbfkv"] Jul 06 20:36:24.366357 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.366340 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666c749768-mbfkv"] Jul 06 20:36:24.366456 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.366431 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.369529 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.369512 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 20:36:24.369621 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.369551 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lqq25\"" Jul 06 20:36:24.369621 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.369594 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 20:36:24.369710 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.369622 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 20:36:24.376184 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.376167 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 20:36:24.420652 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.420601 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:24.420652 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.420631 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln75p\" (UniqueName: \"kubernetes.io/projected/54b52e62-b5cd-4d43-adcf-32649b70e2d6-kube-api-access-ln75p\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:24.420790 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.420689 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/54b52e62-b5cd-4d43-adcf-32649b70e2d6-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:24.420790 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:24.420729 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 20:36:24.420868 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:24.420810 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls podName:54b52e62-b5cd-4d43-adcf-32649b70e2d6 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:24.920795539 +0000 UTC m=+52.006350714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7fkc2" (UID: "54b52e62-b5cd-4d43-adcf-32649b70e2d6") : secret "cluster-monitoring-operator-tls" not found Jul 06 20:36:24.421357 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.421341 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/54b52e62-b5cd-4d43-adcf-32649b70e2d6-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:24.432223 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.432208 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln75p\" (UniqueName: \"kubernetes.io/projected/54b52e62-b5cd-4d43-adcf-32649b70e2d6-kube-api-access-ln75p\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:24.446994 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.446977 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mzdjm"] Jul 06 20:36:24.464269 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.464252 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mzdjm" Jul 06 20:36:24.465724 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.465707 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mzdjm"] Jul 06 20:36:24.467286 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.467265 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mdlqf\"" Jul 06 20:36:24.467389 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.467303 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 06 20:36:24.467389 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.467344 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:36:24.521465 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.521447 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-trusted-ca\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.521560 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.521474 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:24.521560 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.521493 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-installation-pull-secrets\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.521560 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.521508 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-bound-sa-token\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.521560 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.521551 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-ca-trust-extracted\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.521752 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:24.521590 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:36:24.521752 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:24.521627 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls podName:5c07f3a3-0155-4529-a90d-d5dbdacf7b15 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:40.521617659 +0000 UTC m=+67.607172831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls") pod "dns-default-t8qhq" (UID: "5c07f3a3-0155-4529-a90d-d5dbdacf7b15") : secret "dns-default-metrics-tls" not found Jul 06 20:36:24.521752 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.521706 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-certificates\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.521904 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.521775 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-image-registry-private-configuration\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.521904 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.521807 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.521904 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.521833 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qskqw\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-kube-api-access-qskqw\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.623100 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.623077 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-certificates\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.623191 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.623107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-image-registry-private-configuration\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.623191 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.623124 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.623191 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.623139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qskqw\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-kube-api-access-qskqw\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.623320 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.623193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-trusted-ca\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.623320 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:24.623217 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:36:24.623320 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:24.623230 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666c749768-mbfkv: secret "image-registry-tls" not found Jul 06 20:36:24.623320 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.623233 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-installation-pull-secrets\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.623320 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.623254 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-bound-sa-token\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.623320 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:24.623265 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls podName:dea9f7b1-5211-44d3-90c5-a4ec538c8e6a nodeName:}" failed. No retries permitted until 2026-07-06 20:36:25.123252979 +0000 UTC m=+52.208808150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls") pod "image-registry-666c749768-mbfkv" (UID: "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a") : secret "image-registry-tls" not found Jul 06 20:36:24.623320 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.623308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-ca-trust-extracted\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.623633 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.623338 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nq4n\" (UniqueName: \"kubernetes.io/projected/03395407-4e74-4b69-a2df-7f86f312db21-kube-api-access-2nq4n\") pod \"volume-data-source-validator-5d456fdbcd-mzdjm\" (UID: \"03395407-4e74-4b69-a2df-7f86f312db21\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mzdjm" Jul 06 20:36:24.624319 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.624292 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-certificates\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.624416 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.624359 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-ca-trust-extracted\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.626054 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.626021 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-image-registry-private-configuration\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.626799 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.626754 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-installation-pull-secrets\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.628514 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.628493 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-trusted-ca\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.639076 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.639050 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-bound-sa-token\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.639277 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.639260 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qskqw\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-kube-api-access-qskqw\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:24.724516 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.724470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nq4n\" (UniqueName: \"kubernetes.io/projected/03395407-4e74-4b69-a2df-7f86f312db21-kube-api-access-2nq4n\") pod \"volume-data-source-validator-5d456fdbcd-mzdjm\" (UID: \"03395407-4e74-4b69-a2df-7f86f312db21\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mzdjm" Jul 06 20:36:24.734110 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.734087 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nq4n\" (UniqueName: \"kubernetes.io/projected/03395407-4e74-4b69-a2df-7f86f312db21-kube-api-access-2nq4n\") pod \"volume-data-source-validator-5d456fdbcd-mzdjm\" (UID: \"03395407-4e74-4b69-a2df-7f86f312db21\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mzdjm" Jul 06 20:36:24.774207 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.774190 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mzdjm" Jul 06 20:36:24.918422 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.918391 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mzdjm"] Jul 06 20:36:24.921253 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:36:24.921225 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03395407_4e74_4b69_a2df_7f86f312db21.slice/crio-d1969aed0d1736475b2eb3dbe230f482e79d0eb25c92c40b00d9d77db1c2c84c WatchSource:0}: Error finding container d1969aed0d1736475b2eb3dbe230f482e79d0eb25c92c40b00d9d77db1c2c84c: Status 404 returned error can't find the container with id d1969aed0d1736475b2eb3dbe230f482e79d0eb25c92c40b00d9d77db1c2c84c Jul 06 20:36:24.926877 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:24.926852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:24.926989 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:24.926975 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 20:36:24.927052 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:24.927042 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls podName:54b52e62-b5cd-4d43-adcf-32649b70e2d6 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:25.927018414 +0000 UTC m=+53.012573588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7fkc2" (UID: "54b52e62-b5cd-4d43-adcf-32649b70e2d6") : secret "cluster-monitoring-operator-tls" not found Jul 06 20:36:25.128559 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:25.128539 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:25.128701 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:25.128684 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:36:25.128742 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:25.128702 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666c749768-mbfkv: secret "image-registry-tls" not found Jul 06 20:36:25.128799 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:25.128744 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls podName:dea9f7b1-5211-44d3-90c5-a4ec538c8e6a nodeName:}" failed. No retries permitted until 2026-07-06 20:36:26.128731046 +0000 UTC m=+53.214286217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls") pod "image-registry-666c749768-mbfkv" (UID: "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a") : secret "image-registry-tls" not found Jul 06 20:36:25.836864 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:25.836819 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mzdjm" event={"ID":"03395407-4e74-4b69-a2df-7f86f312db21","Type":"ContainerStarted","Data":"d1969aed0d1736475b2eb3dbe230f482e79d0eb25c92c40b00d9d77db1c2c84c"} Jul 06 20:36:25.933458 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:25.933426 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:25.933600 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:25.933569 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 20:36:25.933638 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:25.933630 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls podName:54b52e62-b5cd-4d43-adcf-32649b70e2d6 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:27.933615783 +0000 UTC m=+55.019170954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7fkc2" (UID: "54b52e62-b5cd-4d43-adcf-32649b70e2d6") : secret "cluster-monitoring-operator-tls" not found Jul 06 20:36:26.133972 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:26.133948 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:26.134098 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:26.134078 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:36:26.134098 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:26.134094 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666c749768-mbfkv: secret "image-registry-tls" not found Jul 06 20:36:26.134165 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:26.134139 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls podName:dea9f7b1-5211-44d3-90c5-a4ec538c8e6a nodeName:}" failed. No retries permitted until 2026-07-06 20:36:28.13412569 +0000 UTC m=+55.219680862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls") pod "image-registry-666c749768-mbfkv" (UID: "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a") : secret "image-registry-tls" not found Jul 06 20:36:26.840684 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:26.840655 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mzdjm" event={"ID":"03395407-4e74-4b69-a2df-7f86f312db21","Type":"ContainerStarted","Data":"7f35736f1493fbc280edf42b90e4699802a1489faf5b6cc1ec588c64ec11941b"} Jul 06 20:36:26.859173 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:26.859024 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mzdjm" podStartSLOduration=1.487521363 podStartE2EDuration="2.859006659s" totalImagesPullingTime="1.371485296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:26.858559337 +0000 UTC m=+53.944114529" watchObservedRunningTime="2026-07-06 20:36:26.859006659 +0000 UTC m=+53.944561852" Jul 06 20:36:27.945606 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:27.945573 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:27.945973 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:27.945696 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 20:36:27.945973 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:27.945781 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls podName:54b52e62-b5cd-4d43-adcf-32649b70e2d6 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:31.945742542 +0000 UTC m=+59.031297733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7fkc2" (UID: "54b52e62-b5cd-4d43-adcf-32649b70e2d6") : secret "cluster-monitoring-operator-tls" not found Jul 06 20:36:28.147258 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:28.147229 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:28.147404 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:28.147374 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:36:28.147404 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:28.147393 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666c749768-mbfkv: secret "image-registry-tls" not found Jul 06 20:36:28.147469 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:28.147447 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls podName:dea9f7b1-5211-44d3-90c5-a4ec538c8e6a nodeName:}" failed. No retries permitted until 2026-07-06 20:36:32.147428697 +0000 UTC m=+59.232983868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls") pod "image-registry-666c749768-mbfkv" (UID: "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a") : secret "image-registry-tls" not found Jul 06 20:36:30.128138 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.128110 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lcm6v"] Jul 06 20:36:30.130109 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.130094 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.132815 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.132743 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lh658\"" Jul 06 20:36:30.136969 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.136746 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 20:36:30.136969 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.136779 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 20:36:30.136969 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.136785 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 20:36:30.136969 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.136747 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 20:36:30.141470 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.141451 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lcm6v"] Jul 06 20:36:30.262648 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.262623 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.262748 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.262662 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c111398d-6c9f-4d44-9b46-95fe1fdf4248-data-volume\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.262748 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.262696 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.262748 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.262727 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c111398d-6c9f-4d44-9b46-95fe1fdf4248-crio-socket\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.262878 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.262822 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7jf\" (UniqueName: \"kubernetes.io/projected/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-api-access-gv7jf\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.363947 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.363923 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c111398d-6c9f-4d44-9b46-95fe1fdf4248-crio-socket\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.364036 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.363972 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7jf\" (UniqueName: \"kubernetes.io/projected/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-api-access-gv7jf\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.364036 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.364011 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.364036 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.364034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c111398d-6c9f-4d44-9b46-95fe1fdf4248-data-volume\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.364179 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.364057 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.364179 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:30.364134 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:36:30.364179 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:30.364165 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:36:30.364179 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.364129 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c111398d-6c9f-4d44-9b46-95fe1fdf4248-crio-socket\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.364350 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:30.364200 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm podName:c111398d-6c9f-4d44-9b46-95fe1fdf4248 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:30.864181373 +0000 UTC m=+57.949736544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lcm6v" (UID: "c111398d-6c9f-4d44-9b46-95fe1fdf4248") : configmap "kube-rbac-proxy" not found Jul 06 20:36:30.364350 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:30.364226 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls podName:c111398d-6c9f-4d44-9b46-95fe1fdf4248 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:30.864210236 +0000 UTC m=+57.949765424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcm6v" (UID: "c111398d-6c9f-4d44-9b46-95fe1fdf4248") : secret "insights-runtime-extractor-tls" not found Jul 06 20:36:30.364484 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.364351 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c111398d-6c9f-4d44-9b46-95fe1fdf4248-data-volume\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.376737 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.376705 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7jf\" (UniqueName: \"kubernetes.io/projected/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-api-access-gv7jf\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.706920 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.706896 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-nq9k6"] Jul 06 20:36:30.708874 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.708861 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nq9k6" Jul 06 20:36:30.712033 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.712010 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rx9pk\"" Jul 06 20:36:30.713516 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.713500 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 20:36:30.714966 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.714950 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 20:36:30.724193 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.724168 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-nq9k6"] Jul 06 20:36:30.868490 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.868469 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.868570 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.868511 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:30.868570 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.868532 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfvz7\" (UniqueName: \"kubernetes.io/projected/22500c6c-3d10-49f4-be87-5e9efea17085-kube-api-access-cfvz7\") pod \"migrator-d57d58cc9-nq9k6\" (UID: \"22500c6c-3d10-49f4-be87-5e9efea17085\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nq9k6" Jul 06 20:36:30.868632 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:30.868589 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:36:30.868632 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:30.868607 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:36:30.868692 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:30.868649 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls podName:c111398d-6c9f-4d44-9b46-95fe1fdf4248 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:31.868638753 +0000 UTC m=+58.954193925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcm6v" (UID: "c111398d-6c9f-4d44-9b46-95fe1fdf4248") : secret "insights-runtime-extractor-tls" not found Jul 06 20:36:30.868729 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:30.868720 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm podName:c111398d-6c9f-4d44-9b46-95fe1fdf4248 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:31.868704266 +0000 UTC m=+58.954259448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lcm6v" (UID: "c111398d-6c9f-4d44-9b46-95fe1fdf4248") : configmap "kube-rbac-proxy" not found Jul 06 20:36:30.969707 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.969647 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfvz7\" (UniqueName: \"kubernetes.io/projected/22500c6c-3d10-49f4-be87-5e9efea17085-kube-api-access-cfvz7\") pod \"migrator-d57d58cc9-nq9k6\" (UID: \"22500c6c-3d10-49f4-be87-5e9efea17085\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nq9k6" Jul 06 20:36:30.979397 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:30.979376 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfvz7\" (UniqueName: \"kubernetes.io/projected/22500c6c-3d10-49f4-be87-5e9efea17085-kube-api-access-cfvz7\") pod \"migrator-d57d58cc9-nq9k6\" (UID: \"22500c6c-3d10-49f4-be87-5e9efea17085\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nq9k6" Jul 06 20:36:31.016427 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:31.016406 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nq9k6" Jul 06 20:36:31.156950 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:31.156923 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-nq9k6"] Jul 06 20:36:31.160275 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:36:31.160248 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22500c6c_3d10_49f4_be87_5e9efea17085.slice/crio-411a6a0edc257ce6163235e800edd1a40d4f77a0c069c3e5e5821ee781420bcf WatchSource:0}: Error finding container 411a6a0edc257ce6163235e800edd1a40d4f77a0c069c3e5e5821ee781420bcf: Status 404 returned error can't find the container with id 411a6a0edc257ce6163235e800edd1a40d4f77a0c069c3e5e5821ee781420bcf Jul 06 20:36:31.852139 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:31.852105 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nq9k6" event={"ID":"22500c6c-3d10-49f4-be87-5e9efea17085","Type":"ContainerStarted","Data":"411a6a0edc257ce6163235e800edd1a40d4f77a0c069c3e5e5821ee781420bcf"} Jul 06 20:36:31.876830 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:31.876808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:31.876986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:31.876872 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:31.876986 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:31.876946 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:36:31.877095 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:31.877023 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:36:31.877095 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:31.877028 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm podName:c111398d-6c9f-4d44-9b46-95fe1fdf4248 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:33.87700802 +0000 UTC m=+60.962563199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lcm6v" (UID: "c111398d-6c9f-4d44-9b46-95fe1fdf4248") : configmap "kube-rbac-proxy" not found Jul 06 20:36:31.877095 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:31.877067 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls podName:c111398d-6c9f-4d44-9b46-95fe1fdf4248 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:33.877052965 +0000 UTC m=+60.962608170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcm6v" (UID: "c111398d-6c9f-4d44-9b46-95fe1fdf4248") : secret "insights-runtime-extractor-tls" not found Jul 06 20:36:31.906973 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:31.906956 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nsbpk_e609c626-c0c6-4b01-9640-e63a163c6c96/dns-node-resolver/0.log" Jul 06 20:36:31.977481 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:31.977452 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:31.977618 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:31.977603 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 20:36:31.977690 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:31.977665 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls podName:54b52e62-b5cd-4d43-adcf-32649b70e2d6 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:39.977648751 +0000 UTC m=+67.063203925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7fkc2" (UID: "54b52e62-b5cd-4d43-adcf-32649b70e2d6") : secret "cluster-monitoring-operator-tls" not found Jul 06 20:36:32.179089 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:32.179023 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:32.179398 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:32.179160 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:36:32.179398 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:32.179174 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666c749768-mbfkv: secret "image-registry-tls" not found Jul 06 20:36:32.179398 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:32.179222 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls podName:dea9f7b1-5211-44d3-90c5-a4ec538c8e6a nodeName:}" failed. No retries permitted until 2026-07-06 20:36:40.179209582 +0000 UTC m=+67.264764754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls") pod "image-registry-666c749768-mbfkv" (UID: "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a") : secret "image-registry-tls" not found Jul 06 20:36:32.907629 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:32.907602 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fzclm_a203ceeb-e878-49c1-b1e1-ff93471aa9fb/node-ca/0.log" Jul 06 20:36:33.794753 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:33.794730 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-crszk" Jul 06 20:36:33.860709 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:33.860676 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nq9k6" event={"ID":"22500c6c-3d10-49f4-be87-5e9efea17085","Type":"ContainerStarted","Data":"58ce9b949160f88d8ba06e68acad90ca5a95d428f79b875bfab6e5d71cf311d9"} Jul 06 20:36:33.860879 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:33.860723 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nq9k6" event={"ID":"22500c6c-3d10-49f4-be87-5e9efea17085","Type":"ContainerStarted","Data":"a718d98fcd85e08c77ee54a1b9a37c1f513e254c518ffc851987762c03804b89"} Jul 06 20:36:33.881878 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:33.881837 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-nq9k6" podStartSLOduration=2.015360252 podStartE2EDuration="3.881822835s" totalImagesPullingTime="1.866462583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:33.881593231 +0000 UTC m=+60.967148423" watchObservedRunningTime="2026-07-06 20:36:33.881822835 +0000 UTC m=+60.967378028" Jul 06 20:36:33.891880 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:33.891856 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:33.891990 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:33.891970 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:36:33.892051 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:33.892031 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:33.892051 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:33.892046 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm podName:c111398d-6c9f-4d44-9b46-95fe1fdf4248 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:37.892026295 +0000 UTC m=+64.977581480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lcm6v" (UID: "c111398d-6c9f-4d44-9b46-95fe1fdf4248") : configmap "kube-rbac-proxy" not found Jul 06 20:36:33.892133 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:33.892113 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:36:33.892175 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:33.892160 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls podName:c111398d-6c9f-4d44-9b46-95fe1fdf4248 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:37.892144669 +0000 UTC m=+64.977699858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcm6v" (UID: "c111398d-6c9f-4d44-9b46-95fe1fdf4248") : secret "insights-runtime-extractor-tls" not found Jul 06 20:36:33.908109 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:33.908091 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b7qtk_796f6480-1067-43ca-b1f9-eff19e1a8720/serve-healthcheck-canary/0.log" Jul 06 20:36:37.922559 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:37.922520 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:37.923016 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:37.922576 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:37.923016 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:37.922633 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:36:37.923016 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:37.922659 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:36:37.923016 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:37.922702 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm podName:c111398d-6c9f-4d44-9b46-95fe1fdf4248 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:45.922686597 +0000 UTC m=+73.008241773 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lcm6v" (UID: "c111398d-6c9f-4d44-9b46-95fe1fdf4248") : configmap "kube-rbac-proxy" not found Jul 06 20:36:37.923016 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:37.922715 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls podName:c111398d-6c9f-4d44-9b46-95fe1fdf4248 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:45.922709797 +0000 UTC m=+73.008264968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcm6v" (UID: "c111398d-6c9f-4d44-9b46-95fe1fdf4248") : secret "insights-runtime-extractor-tls" not found Jul 06 20:36:38.223776 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.223692 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:36:38.223776 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.223732 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:36:38.226914 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.226897 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 20:36:38.227260 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.227248 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 20:36:38.234202 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:38.234182 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 20:36:38.234277 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:38.234266 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs podName:4d976ba8-6cb4-4882-9f06-e9c1d62cc475 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:42.234247706 +0000 UTC m=+129.319802883 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs") pod "network-metrics-daemon-d26rt" (UID: "4d976ba8-6cb4-4882-9f06-e9c1d62cc475") : secret "metrics-daemon-secret" not found Jul 06 20:36:38.236235 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.236221 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63a48280-235d-4f85-a741-ff9d9c3aaa3b-original-pull-secret\") pod \"global-pull-secret-syncer-5zpwl\" (UID: \"63a48280-235d-4f85-a741-ff9d9c3aaa3b\") " pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:36:38.324105 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.324074 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n865\" (UniqueName: \"kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865\") pod \"network-check-target-sxdx7\" (UID: \"2fb01258-3ad2-4383-b313-3af4d4a40c54\") " pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:36:38.327322 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.327306 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 20:36:38.338040 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.338020 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 20:36:38.347801 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.347779 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n865\" (UniqueName: \"kubernetes.io/projected/2fb01258-3ad2-4383-b313-3af4d4a40c54-kube-api-access-6n865\") pod \"network-check-target-sxdx7\" (UID: \"2fb01258-3ad2-4383-b313-3af4d4a40c54\") " pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:36:38.446333 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.446310 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c9msn\"" Jul 06 20:36:38.454084 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.454068 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:36:38.455235 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.455219 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zpwl" Jul 06 20:36:38.586159 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.586133 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sxdx7"] Jul 06 20:36:38.589439 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:36:38.589416 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb01258_3ad2_4383_b313_3af4d4a40c54.slice/crio-b0aa1b3f5e008f126fdb89518c1bb8de4eda98c4054f8513f2940e7b1744eef5 WatchSource:0}: Error finding container b0aa1b3f5e008f126fdb89518c1bb8de4eda98c4054f8513f2940e7b1744eef5: Status 404 returned error can't find the container with id b0aa1b3f5e008f126fdb89518c1bb8de4eda98c4054f8513f2940e7b1744eef5 Jul 06 20:36:38.597413 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.597395 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5zpwl"] Jul 06 20:36:38.599481 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:36:38.599461 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a48280_235d_4f85_a741_ff9d9c3aaa3b.slice/crio-917827f9fde59e527df7ab98220772282939c1b34c1033bc0c5c2bfa134693ab WatchSource:0}: Error finding container 917827f9fde59e527df7ab98220772282939c1b34c1033bc0c5c2bfa134693ab: Status 404 returned error can't find the container with id 917827f9fde59e527df7ab98220772282939c1b34c1033bc0c5c2bfa134693ab Jul 06 20:36:38.873243 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.873206 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sxdx7" event={"ID":"2fb01258-3ad2-4383-b313-3af4d4a40c54","Type":"ContainerStarted","Data":"b0aa1b3f5e008f126fdb89518c1bb8de4eda98c4054f8513f2940e7b1744eef5"} Jul 06 20:36:38.874393 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:38.874370 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5zpwl" event={"ID":"63a48280-235d-4f85-a741-ff9d9c3aaa3b","Type":"ContainerStarted","Data":"917827f9fde59e527df7ab98220772282939c1b34c1033bc0c5c2bfa134693ab"} Jul 06 20:36:40.038504 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:40.038468 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:40.038985 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:40.038647 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 20:36:40.038985 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:36:40.038730 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls podName:54b52e62-b5cd-4d43-adcf-32649b70e2d6 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:56.038708416 +0000 UTC m=+83.124263587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7fkc2" (UID: "54b52e62-b5cd-4d43-adcf-32649b70e2d6") : secret "cluster-monitoring-operator-tls" not found Jul 06 20:36:40.241024 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:40.240938 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:40.244820 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:40.244795 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls\") pod \"image-registry-666c749768-mbfkv\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:40.277277 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:40.277247 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lqq25\"" Jul 06 20:36:40.284776 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:40.284737 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:40.451775 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:40.451722 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666c749768-mbfkv"] Jul 06 20:36:40.544022 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:40.543977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:40.547438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:40.547384 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c07f3a3-0155-4529-a90d-d5dbdacf7b15-metrics-tls\") pod \"dns-default-t8qhq\" (UID: \"5c07f3a3-0155-4529-a90d-d5dbdacf7b15\") " pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:40.575451 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:40.575397 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67jtg\"" Jul 06 20:36:40.582990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:40.582960 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:41.476211 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:36:41.476185 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea9f7b1_5211_44d3_90c5_a4ec538c8e6a.slice/crio-946ccb9f373d40c6e06a96b64a56c7e8c885fadca1b94cee6d1ceab379438ece WatchSource:0}: Error finding container 946ccb9f373d40c6e06a96b64a56c7e8c885fadca1b94cee6d1ceab379438ece: Status 404 returned error can't find the container with id 946ccb9f373d40c6e06a96b64a56c7e8c885fadca1b94cee6d1ceab379438ece Jul 06 20:36:41.885779 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:41.885721 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666c749768-mbfkv" event={"ID":"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a","Type":"ContainerStarted","Data":"946ccb9f373d40c6e06a96b64a56c7e8c885fadca1b94cee6d1ceab379438ece"} Jul 06 20:36:43.021508 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:43.021469 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t8qhq"] Jul 06 20:36:43.026595 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:36:43.026566 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c07f3a3_0155_4529_a90d_d5dbdacf7b15.slice/crio-95be559a86361aa22301dc3de6ca118d348eccb6206cffcc07a59e86239b7ad8 WatchSource:0}: Error finding container 95be559a86361aa22301dc3de6ca118d348eccb6206cffcc07a59e86239b7ad8: Status 404 returned error can't find the container with id 95be559a86361aa22301dc3de6ca118d348eccb6206cffcc07a59e86239b7ad8 Jul 06 20:36:43.894198 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:43.894162 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5zpwl" event={"ID":"63a48280-235d-4f85-a741-ff9d9c3aaa3b","Type":"ContainerStarted","Data":"25737c87e91400708c2444aca1c6cb3e15c904ef4ca13e66d999f14c10d5a2cf"} Jul 06 20:36:43.895313 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:43.895286 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t8qhq" event={"ID":"5c07f3a3-0155-4529-a90d-d5dbdacf7b15","Type":"ContainerStarted","Data":"95be559a86361aa22301dc3de6ca118d348eccb6206cffcc07a59e86239b7ad8"} Jul 06 20:36:43.896840 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:43.896810 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sxdx7" event={"ID":"2fb01258-3ad2-4383-b313-3af4d4a40c54","Type":"ContainerStarted","Data":"e1e582230a0ac447789dc769b6868c1795f7802c90aefb15cb994326e2f932a6"} Jul 06 20:36:43.896947 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:43.896910 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:36:43.898278 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:43.898249 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666c749768-mbfkv" event={"ID":"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a","Type":"ContainerStarted","Data":"b03ec099393b71ad6745776f1d16a96289ad9a5f53b03aba28e68c6288a17b59"} Jul 06 20:36:43.898408 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:43.898390 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:36:43.919115 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:43.919065 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5zpwl" podStartSLOduration=65.627194033 podStartE2EDuration="1m9.919048327s" totalImagesPullingTime="4.291854294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:43.91827213 +0000 UTC m=+71.003827335" watchObservedRunningTime="2026-07-06 20:36:43.919048327 +0000 UTC m=+71.004603520" Jul 06 20:36:43.950172 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:43.950127 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sxdx7" podStartSLOduration=65.652625982 podStartE2EDuration="1m9.95011547s" totalImagesPullingTime="4.297489488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:43.949694257 +0000 UTC m=+71.035249451" watchObservedRunningTime="2026-07-06 20:36:43.95011547 +0000 UTC m=+71.035670663" Jul 06 20:36:43.979454 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:43.979413 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-666c749768-mbfkv" podStartSLOduration=19.979403057 podStartE2EDuration="19.979403057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:43.975905367 +0000 UTC m=+71.061460561" watchObservedRunningTime="2026-07-06 20:36:43.979403057 +0000 UTC m=+71.064958249" Jul 06 20:36:44.903902 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:44.903865 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t8qhq" event={"ID":"5c07f3a3-0155-4529-a90d-d5dbdacf7b15","Type":"ContainerStarted","Data":"5d391f2f9c3e3de8148fa83ffa5688131f5b2a2b7e5b0c41b7d9372371f87490"} Jul 06 20:36:44.904260 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:44.903952 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t8qhq" event={"ID":"5c07f3a3-0155-4529-a90d-d5dbdacf7b15","Type":"ContainerStarted","Data":"f59f7f7fb221a7299d5afd954c4742a66c0b0530fccb5240fe416c54301ece77"} Jul 06 20:36:44.904391 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:44.904373 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:44.927804 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:44.927740 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t8qhq" podStartSLOduration=35.43914116 podStartE2EDuration="36.927724548s" totalImagesPullingTime="1.488583388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:44.927136992 +0000 UTC m=+72.012692196" watchObservedRunningTime="2026-07-06 20:36:44.927724548 +0000 UTC m=+72.013279719" Jul 06 20:36:45.988349 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:45.988316 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:45.988684 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:45.988376 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:45.988783 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:45.988750 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c111398d-6c9f-4d44-9b46-95fe1fdf4248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:45.990436 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:45.990419 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c111398d-6c9f-4d44-9b46-95fe1fdf4248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcm6v\" (UID: \"c111398d-6c9f-4d44-9b46-95fe1fdf4248\") " pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:46.046733 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:46.046710 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lh658\"" Jul 06 20:36:46.052824 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:46.052809 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lcm6v" Jul 06 20:36:46.178263 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:46.178238 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lcm6v"] Jul 06 20:36:46.182119 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:36:46.182089 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc111398d_6c9f_4d44_9b46_95fe1fdf4248.slice/crio-db521ba62d934651df4248208e08977a51b8e45f8d5c46d57287dfcc95751058 WatchSource:0}: Error finding container db521ba62d934651df4248208e08977a51b8e45f8d5c46d57287dfcc95751058: Status 404 returned error can't find the container with id db521ba62d934651df4248208e08977a51b8e45f8d5c46d57287dfcc95751058 Jul 06 20:36:46.914550 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:46.914468 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lcm6v" event={"ID":"c111398d-6c9f-4d44-9b46-95fe1fdf4248","Type":"ContainerStarted","Data":"db521ba62d934651df4248208e08977a51b8e45f8d5c46d57287dfcc95751058"} Jul 06 20:36:49.925165 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:49.925129 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lcm6v" event={"ID":"c111398d-6c9f-4d44-9b46-95fe1fdf4248","Type":"ContainerStarted","Data":"b5e72c74e75867803b43caee996419cc81777f0c9817ca9d19106f74e3906a7e"} Jul 06 20:36:50.931003 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:50.930966 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lcm6v" event={"ID":"c111398d-6c9f-4d44-9b46-95fe1fdf4248","Type":"ContainerStarted","Data":"2eef4f2dc541c46dc697482ee75bd13f9fc5f6119be714c8b27cd576901d3f79"} Jul 06 20:36:52.939273 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:52.939210 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lcm6v" event={"ID":"c111398d-6c9f-4d44-9b46-95fe1fdf4248","Type":"ContainerStarted","Data":"33108f38b957defdc8139c0d680ff19fff25468944c7a1b81772fc73ee83d193"} Jul 06 20:36:52.961835 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:52.961787 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lcm6v" podStartSLOduration=16.647748909 podStartE2EDuration="22.961772468s" totalImagesPullingTime="6.314023559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:36:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:52.960658166 +0000 UTC m=+80.046213362" watchObservedRunningTime="2026-07-06 20:36:52.961772468 +0000 UTC m=+80.047327651" Jul 06 20:36:54.502911 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.502881 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j"] Jul 06 20:36:54.505086 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.505065 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" Jul 06 20:36:54.510058 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.510024 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 20:36:54.510171 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.510043 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 20:36:54.510171 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.510047 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 20:36:54.510362 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.510347 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 20:36:54.523772 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.523735 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666c749768-mbfkv"] Jul 06 20:36:54.524799 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.524780 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j"] Jul 06 20:36:54.550418 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.550399 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ffc2142-4f78-450c-bbfe-c7db04b949c6-tmp\") pod \"klusterlet-addon-workmgr-df86c5d5-h8v9j\" (UID: \"0ffc2142-4f78-450c-bbfe-c7db04b949c6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" Jul 06 20:36:54.550507 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.550446 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdwg\" (UniqueName: \"kubernetes.io/projected/0ffc2142-4f78-450c-bbfe-c7db04b949c6-kube-api-access-skdwg\") pod \"klusterlet-addon-workmgr-df86c5d5-h8v9j\" (UID: \"0ffc2142-4f78-450c-bbfe-c7db04b949c6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" Jul 06 20:36:54.550507 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.550501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0ffc2142-4f78-450c-bbfe-c7db04b949c6-klusterlet-config\") pod \"klusterlet-addon-workmgr-df86c5d5-h8v9j\" (UID: \"0ffc2142-4f78-450c-bbfe-c7db04b949c6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" Jul 06 20:36:54.599142 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.599117 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-gvh6p"] Jul 06 20:36:54.601075 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.601060 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-gvh6p" Jul 06 20:36:54.603595 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.603575 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr"] Jul 06 20:36:54.604450 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.604433 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fjcbd\"" Jul 06 20:36:54.605392 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.605377 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 20:36:54.605475 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.605404 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 20:36:54.605567 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.605551 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr" Jul 06 20:36:54.608347 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.608326 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 20:36:54.608446 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.608359 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fqgcb\"" Jul 06 20:36:54.609100 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.609085 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 20:36:54.623230 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.623210 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-gvh6p"] Jul 06 20:36:54.626074 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.626055 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr"] Jul 06 20:36:54.651572 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.651550 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0ffc2142-4f78-450c-bbfe-c7db04b949c6-klusterlet-config\") pod \"klusterlet-addon-workmgr-df86c5d5-h8v9j\" (UID: \"0ffc2142-4f78-450c-bbfe-c7db04b949c6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" Jul 06 20:36:54.651662 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.651582 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ffc2142-4f78-450c-bbfe-c7db04b949c6-tmp\") pod \"klusterlet-addon-workmgr-df86c5d5-h8v9j\" (UID: \"0ffc2142-4f78-450c-bbfe-c7db04b949c6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" Jul 06 20:36:54.651662 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.651617 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a472ef31-0009-4bc8-97f2-72ce4847739d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-p4fbr\" (UID: \"a472ef31-0009-4bc8-97f2-72ce4847739d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr" Jul 06 20:36:54.651662 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.651643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a472ef31-0009-4bc8-97f2-72ce4847739d-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-p4fbr\" (UID: \"a472ef31-0009-4bc8-97f2-72ce4847739d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr" Jul 06 20:36:54.651846 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.651689 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdwg\" (UniqueName: \"kubernetes.io/projected/0ffc2142-4f78-450c-bbfe-c7db04b949c6-kube-api-access-skdwg\") pod \"klusterlet-addon-workmgr-df86c5d5-h8v9j\" (UID: \"0ffc2142-4f78-450c-bbfe-c7db04b949c6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" Jul 06 20:36:54.651846 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.651754 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48kq7\" (UniqueName: \"kubernetes.io/projected/9f0b356c-3738-4e8e-a2f4-8b159924540a-kube-api-access-48kq7\") pod \"downloads-685bfb5f64-gvh6p\" (UID: \"9f0b356c-3738-4e8e-a2f4-8b159924540a\") " pod="openshift-console/downloads-685bfb5f64-gvh6p" Jul 06 20:36:54.651990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.651973 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ffc2142-4f78-450c-bbfe-c7db04b949c6-tmp\") pod \"klusterlet-addon-workmgr-df86c5d5-h8v9j\" (UID: \"0ffc2142-4f78-450c-bbfe-c7db04b949c6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" Jul 06 20:36:54.653860 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.653846 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0ffc2142-4f78-450c-bbfe-c7db04b949c6-klusterlet-config\") pod \"klusterlet-addon-workmgr-df86c5d5-h8v9j\" (UID: \"0ffc2142-4f78-450c-bbfe-c7db04b949c6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" Jul 06 20:36:54.668142 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.668125 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdwg\" (UniqueName: \"kubernetes.io/projected/0ffc2142-4f78-450c-bbfe-c7db04b949c6-kube-api-access-skdwg\") pod \"klusterlet-addon-workmgr-df86c5d5-h8v9j\" (UID: \"0ffc2142-4f78-450c-bbfe-c7db04b949c6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" Jul 06 20:36:54.752810 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.752793 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48kq7\" (UniqueName: \"kubernetes.io/projected/9f0b356c-3738-4e8e-a2f4-8b159924540a-kube-api-access-48kq7\") pod \"downloads-685bfb5f64-gvh6p\" (UID: \"9f0b356c-3738-4e8e-a2f4-8b159924540a\") " pod="openshift-console/downloads-685bfb5f64-gvh6p" Jul 06 20:36:54.752943 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.752835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a472ef31-0009-4bc8-97f2-72ce4847739d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-p4fbr\" (UID: \"a472ef31-0009-4bc8-97f2-72ce4847739d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr" Jul 06 20:36:54.752943 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.752852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a472ef31-0009-4bc8-97f2-72ce4847739d-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-p4fbr\" (UID: \"a472ef31-0009-4bc8-97f2-72ce4847739d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr" Jul 06 20:36:54.753345 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.753329 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a472ef31-0009-4bc8-97f2-72ce4847739d-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-p4fbr\" (UID: \"a472ef31-0009-4bc8-97f2-72ce4847739d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr" Jul 06 20:36:54.754896 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.754877 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a472ef31-0009-4bc8-97f2-72ce4847739d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-p4fbr\" (UID: \"a472ef31-0009-4bc8-97f2-72ce4847739d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr" Jul 06 20:36:54.763231 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.763210 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48kq7\" (UniqueName: \"kubernetes.io/projected/9f0b356c-3738-4e8e-a2f4-8b159924540a-kube-api-access-48kq7\") pod \"downloads-685bfb5f64-gvh6p\" (UID: \"9f0b356c-3738-4e8e-a2f4-8b159924540a\") " pod="openshift-console/downloads-685bfb5f64-gvh6p" Jul 06 20:36:54.813952 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.813929 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" Jul 06 20:36:54.910832 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.910807 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-gvh6p" Jul 06 20:36:54.915991 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.915969 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr" Jul 06 20:36:54.938038 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:54.938005 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j"] Jul 06 20:36:54.942930 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:36:54.942907 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ffc2142_4f78_450c_bbfe_c7db04b949c6.slice/crio-99f182277aa745a450071543216579a08b1ea35ee8239bf814f29eb146eadc64 WatchSource:0}: Error finding container 99f182277aa745a450071543216579a08b1ea35ee8239bf814f29eb146eadc64: Status 404 returned error can't find the container with id 99f182277aa745a450071543216579a08b1ea35ee8239bf814f29eb146eadc64 Jul 06 20:36:55.041259 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:55.041235 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-gvh6p"] Jul 06 20:36:55.044042 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:36:55.044014 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0b356c_3738_4e8e_a2f4_8b159924540a.slice/crio-e70ed2a7d63ac66a2d13b93f3401a1e5fcf00454f555e4a0bf2eb750efc71803 WatchSource:0}: Error finding container e70ed2a7d63ac66a2d13b93f3401a1e5fcf00454f555e4a0bf2eb750efc71803: Status 404 returned error can't find the container with id e70ed2a7d63ac66a2d13b93f3401a1e5fcf00454f555e4a0bf2eb750efc71803 Jul 06 20:36:55.064148 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:55.064127 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr"] Jul 06 20:36:55.066634 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:36:55.066615 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda472ef31_0009_4bc8_97f2_72ce4847739d.slice/crio-2f217bc650bba65aaf7939956d4e21208eed4213236ec167de0097fd8acd8902 WatchSource:0}: Error finding container 2f217bc650bba65aaf7939956d4e21208eed4213236ec167de0097fd8acd8902: Status 404 returned error can't find the container with id 2f217bc650bba65aaf7939956d4e21208eed4213236ec167de0097fd8acd8902 Jul 06 20:36:55.917334 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:55.917295 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t8qhq" Jul 06 20:36:55.953190 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:55.953156 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr" event={"ID":"a472ef31-0009-4bc8-97f2-72ce4847739d","Type":"ContainerStarted","Data":"2f217bc650bba65aaf7939956d4e21208eed4213236ec167de0097fd8acd8902"} Jul 06 20:36:55.957682 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:55.957499 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-gvh6p" event={"ID":"9f0b356c-3738-4e8e-a2f4-8b159924540a","Type":"ContainerStarted","Data":"e70ed2a7d63ac66a2d13b93f3401a1e5fcf00454f555e4a0bf2eb750efc71803"} Jul 06 20:36:55.960026 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:55.959098 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" event={"ID":"0ffc2142-4f78-450c-bbfe-c7db04b949c6","Type":"ContainerStarted","Data":"99f182277aa745a450071543216579a08b1ea35ee8239bf814f29eb146eadc64"} Jul 06 20:36:56.069681 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:56.063465 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:56.075115 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:56.075059 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54b52e62-b5cd-4d43-adcf-32649b70e2d6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7fkc2\" (UID: \"54b52e62-b5cd-4d43-adcf-32649b70e2d6\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:56.273039 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:56.272950 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qcxw2\"" Jul 06 20:36:56.280422 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:56.280378 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" Jul 06 20:36:57.274629 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:57.274599 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2"] Jul 06 20:36:57.277831 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:36:57.277805 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b52e62_b5cd_4d43_adcf_32649b70e2d6.slice/crio-3ddf81bc234cd5f0d1b7c79f575a5dfc370e4d250a008285ccd318de29fb212a WatchSource:0}: Error finding container 3ddf81bc234cd5f0d1b7c79f575a5dfc370e4d250a008285ccd318de29fb212a: Status 404 returned error can't find the container with id 3ddf81bc234cd5f0d1b7c79f575a5dfc370e4d250a008285ccd318de29fb212a Jul 06 20:36:57.967140 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:57.967103 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr" event={"ID":"a472ef31-0009-4bc8-97f2-72ce4847739d","Type":"ContainerStarted","Data":"67300e23e3da6bed5d7a7f60321c9f1986fcbe52fa503aa732df3dcc9f5facbb"} Jul 06 20:36:57.968213 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:57.968190 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" event={"ID":"54b52e62-b5cd-4d43-adcf-32649b70e2d6","Type":"ContainerStarted","Data":"3ddf81bc234cd5f0d1b7c79f575a5dfc370e4d250a008285ccd318de29fb212a"} Jul 06 20:36:57.992915 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:57.992863 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-p4fbr" podStartSLOduration=1.930701469 podStartE2EDuration="3.99284628s" totalImagesPullingTime="2.062144811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:57.991544444 +0000 UTC m=+85.077099637" watchObservedRunningTime="2026-07-06 20:36:57.99284628 +0000 UTC m=+85.078401474" Jul 06 20:36:59.663581 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:59.663546 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtgmv"] Jul 06 20:36:59.665874 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:59.665852 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtgmv" Jul 06 20:36:59.668548 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:59.668530 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lvfq8\"" Jul 06 20:36:59.668645 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:59.668626 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 20:36:59.675979 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:59.675960 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtgmv"] Jul 06 20:36:59.690432 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:59.690409 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a81bf929-9dbc-4714-8ace-3c27e9c96b30-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-qtgmv\" (UID: \"a81bf929-9dbc-4714-8ace-3c27e9c96b30\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtgmv" Jul 06 20:36:59.790953 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:59.790931 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a81bf929-9dbc-4714-8ace-3c27e9c96b30-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-qtgmv\" (UID: \"a81bf929-9dbc-4714-8ace-3c27e9c96b30\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtgmv" Jul 06 20:36:59.793113 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:59.793088 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a81bf929-9dbc-4714-8ace-3c27e9c96b30-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-qtgmv\" (UID: \"a81bf929-9dbc-4714-8ace-3c27e9c96b30\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtgmv" Jul 06 20:36:59.974287 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:59.974217 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtgmv" Jul 06 20:36:59.981094 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:36:59.981067 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" event={"ID":"54b52e62-b5cd-4d43-adcf-32649b70e2d6","Type":"ContainerStarted","Data":"dfc6bfd7f86e53fec558311911457d0b10e32e60e8524a57c40e72027dfa3c6d"} Jul 06 20:37:00.001902 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:00.001846 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7fkc2" podStartSLOduration=34.14836714 podStartE2EDuration="36.001832052s" totalImagesPullingTime="1.853464912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:00.001569338 +0000 UTC m=+87.087124532" watchObservedRunningTime="2026-07-06 20:37:00.001832052 +0000 UTC m=+87.087387244" Jul 06 20:37:00.143497 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:00.143468 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtgmv"] Jul 06 20:37:00.146388 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:37:00.146358 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81bf929_9dbc_4714_8ace_3c27e9c96b30.slice/crio-a7b9453b21abef1baf30c646703b03daeb59e7c3cacb04b022f3f2c0a9a9c5e5 WatchSource:0}: Error finding container a7b9453b21abef1baf30c646703b03daeb59e7c3cacb04b022f3f2c0a9a9c5e5: Status 404 returned error can't find the container with id a7b9453b21abef1baf30c646703b03daeb59e7c3cacb04b022f3f2c0a9a9c5e5 Jul 06 20:37:00.986786 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:00.986708 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtgmv" event={"ID":"a81bf929-9dbc-4714-8ace-3c27e9c96b30","Type":"ContainerStarted","Data":"a7b9453b21abef1baf30c646703b03daeb59e7c3cacb04b022f3f2c0a9a9c5e5"} Jul 06 20:37:01.991615 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:01.991579 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtgmv" event={"ID":"a81bf929-9dbc-4714-8ace-3c27e9c96b30","Type":"ContainerStarted","Data":"110120f5871d29bdcf5eefaa455594977828dbf0100627103dc8c25a63cd93ec"} Jul 06 20:37:01.992044 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:01.991794 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtgmv" Jul 06 20:37:01.996941 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:01.996918 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtgmv" Jul 06 20:37:02.018641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.018515 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtgmv" podStartSLOduration=1.985497963 podStartE2EDuration="3.018498007s" totalImagesPullingTime="1.033000044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:02.018495675 +0000 UTC m=+89.104050868" watchObservedRunningTime="2026-07-06 20:37:02.018498007 +0000 UTC m=+89.104053204" Jul 06 20:37:02.737305 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.737271 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl"] Jul 06 20:37:02.739402 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.739385 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:02.742210 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.742185 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 20:37:02.742434 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.742413 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 20:37:02.743503 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.743453 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-br7kh\"" Jul 06 20:37:02.743751 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.743732 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 20:37:02.752583 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.752530 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl"] Jul 06 20:37:02.812587 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.812566 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvpdk\" (UniqueName: \"kubernetes.io/projected/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-kube-api-access-bvpdk\") pod \"prometheus-operator-567f7b8b7c-zc2gl\" (UID: \"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:02.812700 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.812632 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-zc2gl\" (UID: \"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:02.812750 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.812719 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-zc2gl\" (UID: \"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:02.812820 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.812781 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-zc2gl\" (UID: \"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:02.913802 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.913746 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-zc2gl\" (UID: \"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:02.913894 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.913810 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvpdk\" (UniqueName: \"kubernetes.io/projected/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-kube-api-access-bvpdk\") pod \"prometheus-operator-567f7b8b7c-zc2gl\" (UID: \"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:02.913894 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.913866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-zc2gl\" (UID: \"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:02.914031 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.913938 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-zc2gl\" (UID: \"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:02.914080 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:37:02.914033 2582 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 06 20:37:02.914124 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:37:02.914084 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-prometheus-operator-tls podName:a8f6bf9b-d82a-4800-9024-9ba87b46f2fb nodeName:}" failed. No retries permitted until 2026-07-06 20:37:03.41406981 +0000 UTC m=+90.499624981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-zc2gl" (UID: "a8f6bf9b-d82a-4800-9024-9ba87b46f2fb") : secret "prometheus-operator-tls" not found Jul 06 20:37:02.914450 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.914432 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-zc2gl\" (UID: \"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:02.916053 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.916034 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-zc2gl\" (UID: \"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:02.926245 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:02.926223 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvpdk\" (UniqueName: \"kubernetes.io/projected/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-kube-api-access-bvpdk\") pod \"prometheus-operator-567f7b8b7c-zc2gl\" (UID: \"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:03.416845 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:03.416811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-zc2gl\" (UID: \"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:03.419271 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:03.419246 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8f6bf9b-d82a-4800-9024-9ba87b46f2fb-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-zc2gl\" (UID: \"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:03.649719 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:03.649688 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" Jul 06 20:37:03.793389 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:03.793238 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl"] Jul 06 20:37:03.796026 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:37:03.795997 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f6bf9b_d82a_4800_9024_9ba87b46f2fb.slice/crio-5925176653f47380d2e9073f45572046eb4d81f8f4db9b41bae47c800f865efb WatchSource:0}: Error finding container 5925176653f47380d2e9073f45572046eb4d81f8f4db9b41bae47c800f865efb: Status 404 returned error can't find the container with id 5925176653f47380d2e9073f45572046eb4d81f8f4db9b41bae47c800f865efb Jul 06 20:37:03.999995 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:03.999927 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" event={"ID":"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb","Type":"ContainerStarted","Data":"5925176653f47380d2e9073f45572046eb4d81f8f4db9b41bae47c800f865efb"} Jul 06 20:37:04.528874 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:04.528851 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:37:06.010722 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:06.010685 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" event={"ID":"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb","Type":"ContainerStarted","Data":"57e81c2d6fbfeab15f12f274d5050a2093a976d992651308903a85a410d7b4d4"} Jul 06 20:37:06.011182 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:06.010748 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" event={"ID":"a8f6bf9b-d82a-4800-9024-9ba87b46f2fb","Type":"ContainerStarted","Data":"fbc9cf0de58af4a88923ecd5c3349713e93c61f1e05568f4c38dacc72542b111"} Jul 06 20:37:06.042516 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:06.042477 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-zc2gl" podStartSLOduration=2.67087163 podStartE2EDuration="4.042462875s" totalImagesPullingTime="1.371591245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:06.042274756 +0000 UTC m=+93.127829948" watchObservedRunningTime="2026-07-06 20:37:06.042462875 +0000 UTC m=+93.128018067" Jul 06 20:37:08.143390 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.143358 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj"] Jul 06 20:37:08.147159 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.147137 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.151356 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.151329 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 20:37:08.151465 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.151328 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 20:37:08.151465 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.151382 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 20:37:08.151465 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.151388 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xjxtw\"" Jul 06 20:37:08.161949 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.161909 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj"] Jul 06 20:37:08.179048 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.179025 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7bv7x"] Jul 06 20:37:08.182311 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.182287 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.185611 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.185573 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 20:37:08.185694 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.185634 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 20:37:08.185694 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.185574 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 20:37:08.185694 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.185685 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-57hfd\"" Jul 06 20:37:08.257259 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257225 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrdm\" (UniqueName: \"kubernetes.io/projected/91ad3522-11b2-4d4d-b6a5-d913f4b24933-kube-api-access-jsrdm\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.257398 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257264 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c366d3eb-8e1c-4934-9039-50bc149983eb-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.257398 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257296 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ad3522-11b2-4d4d-b6a5-d913f4b24933-metrics-client-ca\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.257398 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257392 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c366d3eb-8e1c-4934-9039-50bc149983eb-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.257540 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257411 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njnqc\" (UniqueName: \"kubernetes.io/projected/c366d3eb-8e1c-4934-9039-50bc149983eb-kube-api-access-njnqc\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.257540 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257443 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c366d3eb-8e1c-4934-9039-50bc149983eb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.257540 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257461 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.257540 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257477 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c366d3eb-8e1c-4934-9039-50bc149983eb-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.257540 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257498 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91ad3522-11b2-4d4d-b6a5-d913f4b24933-root\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.257718 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257574 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-textfile\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.257718 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257637 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-wtmp\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.257718 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257668 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.257718 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257693 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91ad3522-11b2-4d4d-b6a5-d913f4b24933-sys\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.257899 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257723 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c366d3eb-8e1c-4934-9039-50bc149983eb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.257899 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.257784 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-tls\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.358950 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.358919 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-wtmp\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.358950 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.358955 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.359180 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.358979 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91ad3522-11b2-4d4d-b6a5-d913f4b24933-sys\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.359180 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359005 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c366d3eb-8e1c-4934-9039-50bc149983eb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.359180 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359030 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-tls\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.359180 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359065 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrdm\" (UniqueName: \"kubernetes.io/projected/91ad3522-11b2-4d4d-b6a5-d913f4b24933-kube-api-access-jsrdm\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.359180 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359076 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91ad3522-11b2-4d4d-b6a5-d913f4b24933-sys\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.359180 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359091 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c366d3eb-8e1c-4934-9039-50bc149983eb-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.359180 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359120 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-wtmp\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.359180 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359157 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ad3522-11b2-4d4d-b6a5-d913f4b24933-metrics-client-ca\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.359542 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359197 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c366d3eb-8e1c-4934-9039-50bc149983eb-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.359542 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359224 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njnqc\" (UniqueName: \"kubernetes.io/projected/c366d3eb-8e1c-4934-9039-50bc149983eb-kube-api-access-njnqc\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.359542 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359253 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c366d3eb-8e1c-4934-9039-50bc149983eb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.359542 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359282 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.359542 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359310 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c366d3eb-8e1c-4934-9039-50bc149983eb-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.359542 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91ad3522-11b2-4d4d-b6a5-d913f4b24933-root\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.359542 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359375 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-textfile\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.359542 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359480 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c366d3eb-8e1c-4934-9039-50bc149983eb-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.359932 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.359866 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91ad3522-11b2-4d4d-b6a5-d913f4b24933-root\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.360182 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.360154 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ad3522-11b2-4d4d-b6a5-d913f4b24933-metrics-client-ca\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.360182 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.360173 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c366d3eb-8e1c-4934-9039-50bc149983eb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.360347 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.360180 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c366d3eb-8e1c-4934-9039-50bc149983eb-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.360347 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.360329 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-textfile\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.360724 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.360697 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.361777 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.361742 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.361932 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.361897 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91ad3522-11b2-4d4d-b6a5-d913f4b24933-node-exporter-tls\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.362303 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.362260 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c366d3eb-8e1c-4934-9039-50bc149983eb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.362512 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.362460 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c366d3eb-8e1c-4934-9039-50bc149983eb-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.368705 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.368687 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrdm\" (UniqueName: \"kubernetes.io/projected/91ad3522-11b2-4d4d-b6a5-d913f4b24933-kube-api-access-jsrdm\") pod \"node-exporter-7bv7x\" (UID: \"91ad3522-11b2-4d4d-b6a5-d913f4b24933\") " pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.371348 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.371326 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njnqc\" (UniqueName: \"kubernetes.io/projected/c366d3eb-8e1c-4934-9039-50bc149983eb-kube-api-access-njnqc\") pod \"kube-state-metrics-7c9dbb8b7d-rxxbj\" (UID: \"c366d3eb-8e1c-4934-9039-50bc149983eb\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.458368 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.458287 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" Jul 06 20:37:08.492490 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.492467 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7bv7x" Jul 06 20:37:08.857177 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.857131 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b96775d75-sw4cx"] Jul 06 20:37:08.860540 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.860517 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:08.863661 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.863641 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qzqw5\"" Jul 06 20:37:08.863746 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.863670 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 20:37:08.863746 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.863717 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 20:37:08.864933 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.864901 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 20:37:08.865018 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.864912 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 20:37:08.865242 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.865223 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 20:37:08.869824 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.869803 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 20:37:08.873421 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.873400 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b96775d75-sw4cx"] Jul 06 20:37:08.965043 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.965019 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-oauth-config\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:08.965183 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.965053 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-serving-cert\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:08.965183 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.965108 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-oauth-serving-cert\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:08.965183 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.965140 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-service-ca\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:08.965183 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.965169 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbbrh\" (UniqueName: \"kubernetes.io/projected/6bcd74e1-add1-4329-a05b-89be4d575b0d-kube-api-access-gbbrh\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:08.965342 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.965214 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-trusted-ca-bundle\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:08.965342 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:08.965253 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-config\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.066824 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.066147 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-serving-cert\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.066824 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.066212 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-oauth-serving-cert\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.066824 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.066267 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-service-ca\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.066824 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.066299 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbbrh\" (UniqueName: \"kubernetes.io/projected/6bcd74e1-add1-4329-a05b-89be4d575b0d-kube-api-access-gbbrh\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.066824 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.066329 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-trusted-ca-bundle\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.066824 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.066359 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-config\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.066824 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.066396 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-oauth-config\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.068821 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.067609 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-oauth-serving-cert\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.068821 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.068176 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-service-ca\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.068821 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.068735 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-config\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.069062 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.068820 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-trusted-ca-bundle\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.070861 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.070817 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-oauth-config\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.071984 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.071954 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-serving-cert\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.081701 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.081657 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbbrh\" (UniqueName: \"kubernetes.io/projected/6bcd74e1-add1-4329-a05b-89be4d575b0d-kube-api-access-gbbrh\") pod \"console-7b96775d75-sw4cx\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:09.173186 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:09.173099 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:12.578379 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.578340 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f6567c9b6-d6gt6"] Jul 06 20:37:12.706675 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.706593 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f6567c9b6-d6gt6"] Jul 06 20:37:12.706879 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.706748 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.710957 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.710928 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 20:37:12.711067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.710977 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 20:37:12.711158 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.711132 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2nu1ak9scvp12\"" Jul 06 20:37:12.711246 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.711224 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 20:37:12.711246 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.711232 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 20:37:12.711246 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.711232 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rpw7s\"" Jul 06 20:37:12.797617 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.797590 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/194f7868-6ebd-4220-a09d-d21f124c5c2d-metrics-server-audit-profiles\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.797735 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.797636 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/194f7868-6ebd-4220-a09d-d21f124c5c2d-secret-metrics-server-tls\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.797735 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.797665 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv2nx\" (UniqueName: \"kubernetes.io/projected/194f7868-6ebd-4220-a09d-d21f124c5c2d-kube-api-access-qv2nx\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.797735 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.797711 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/194f7868-6ebd-4220-a09d-d21f124c5c2d-client-ca-bundle\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.797735 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.797730 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/194f7868-6ebd-4220-a09d-d21f124c5c2d-audit-log\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.797938 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.797782 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/194f7868-6ebd-4220-a09d-d21f124c5c2d-secret-metrics-server-client-certs\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.797938 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.797855 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/194f7868-6ebd-4220-a09d-d21f124c5c2d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.877926 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.877895 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b"] Jul 06 20:37:12.899265 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.899242 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv2nx\" (UniqueName: \"kubernetes.io/projected/194f7868-6ebd-4220-a09d-d21f124c5c2d-kube-api-access-qv2nx\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.899420 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.899279 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/194f7868-6ebd-4220-a09d-d21f124c5c2d-client-ca-bundle\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.899420 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.899307 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/194f7868-6ebd-4220-a09d-d21f124c5c2d-audit-log\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.899420 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.899333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/194f7868-6ebd-4220-a09d-d21f124c5c2d-secret-metrics-server-client-certs\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.899420 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.899362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/194f7868-6ebd-4220-a09d-d21f124c5c2d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.899420 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.899414 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/194f7868-6ebd-4220-a09d-d21f124c5c2d-metrics-server-audit-profiles\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.899675 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.899461 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/194f7868-6ebd-4220-a09d-d21f124c5c2d-secret-metrics-server-tls\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.899837 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.899786 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/194f7868-6ebd-4220-a09d-d21f124c5c2d-audit-log\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.900215 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.900167 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/194f7868-6ebd-4220-a09d-d21f124c5c2d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.900617 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.900595 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/194f7868-6ebd-4220-a09d-d21f124c5c2d-metrics-server-audit-profiles\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.902331 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.902298 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/194f7868-6ebd-4220-a09d-d21f124c5c2d-client-ca-bundle\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.902442 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.902316 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/194f7868-6ebd-4220-a09d-d21f124c5c2d-secret-metrics-server-tls\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.902442 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.902361 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/194f7868-6ebd-4220-a09d-d21f124c5c2d-secret-metrics-server-client-certs\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.906772 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.906737 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b"] Jul 06 20:37:12.906868 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.906855 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" Jul 06 20:37:12.909118 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.909098 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv2nx\" (UniqueName: \"kubernetes.io/projected/194f7868-6ebd-4220-a09d-d21f124c5c2d-kube-api-access-qv2nx\") pod \"metrics-server-5f6567c9b6-d6gt6\" (UID: \"194f7868-6ebd-4220-a09d-d21f124c5c2d\") " pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:12.909810 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.909789 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 20:37:12.909810 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:12.909808 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vh74f\"" Jul 06 20:37:13.000718 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:13.000685 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83cba701-7e6b-4e88-a4fb-2d984b3fa9b4-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-ff52b\" (UID: \"83cba701-7e6b-4e88-a4fb-2d984b3fa9b4\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" Jul 06 20:37:13.000863 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:13.000740 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x8q9\" (UniqueName: \"kubernetes.io/projected/83cba701-7e6b-4e88-a4fb-2d984b3fa9b4-kube-api-access-8x8q9\") pod \"monitoring-plugin-d5dcff7cf-ff52b\" (UID: \"83cba701-7e6b-4e88-a4fb-2d984b3fa9b4\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" Jul 06 20:37:13.017718 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:13.017693 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:13.102112 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:13.102081 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x8q9\" (UniqueName: \"kubernetes.io/projected/83cba701-7e6b-4e88-a4fb-2d984b3fa9b4-kube-api-access-8x8q9\") pod \"monitoring-plugin-d5dcff7cf-ff52b\" (UID: \"83cba701-7e6b-4e88-a4fb-2d984b3fa9b4\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" Jul 06 20:37:13.102247 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:13.102188 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83cba701-7e6b-4e88-a4fb-2d984b3fa9b4-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-ff52b\" (UID: \"83cba701-7e6b-4e88-a4fb-2d984b3fa9b4\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" Jul 06 20:37:13.102323 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:37:13.102300 2582 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 06 20:37:13.102416 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:37:13.102374 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83cba701-7e6b-4e88-a4fb-2d984b3fa9b4-monitoring-plugin-cert podName:83cba701-7e6b-4e88-a4fb-2d984b3fa9b4 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:13.602356262 +0000 UTC m=+100.687911439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/83cba701-7e6b-4e88-a4fb-2d984b3fa9b4-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-ff52b" (UID: "83cba701-7e6b-4e88-a4fb-2d984b3fa9b4") : secret "monitoring-plugin-cert" not found Jul 06 20:37:13.115168 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:13.115139 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x8q9\" (UniqueName: \"kubernetes.io/projected/83cba701-7e6b-4e88-a4fb-2d984b3fa9b4-kube-api-access-8x8q9\") pod \"monitoring-plugin-d5dcff7cf-ff52b\" (UID: \"83cba701-7e6b-4e88-a4fb-2d984b3fa9b4\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" Jul 06 20:37:13.606400 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:13.606367 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83cba701-7e6b-4e88-a4fb-2d984b3fa9b4-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-ff52b\" (UID: \"83cba701-7e6b-4e88-a4fb-2d984b3fa9b4\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" Jul 06 20:37:13.606804 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:37:13.606551 2582 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 06 20:37:13.606804 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:37:13.606628 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83cba701-7e6b-4e88-a4fb-2d984b3fa9b4-monitoring-plugin-cert podName:83cba701-7e6b-4e88-a4fb-2d984b3fa9b4 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:14.606608467 +0000 UTC m=+101.692163643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/83cba701-7e6b-4e88-a4fb-2d984b3fa9b4-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-ff52b" (UID: "83cba701-7e6b-4e88-a4fb-2d984b3fa9b4") : secret "monitoring-plugin-cert" not found Jul 06 20:37:14.617156 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:14.617127 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83cba701-7e6b-4e88-a4fb-2d984b3fa9b4-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-ff52b\" (UID: \"83cba701-7e6b-4e88-a4fb-2d984b3fa9b4\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" Jul 06 20:37:14.619563 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:14.619532 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83cba701-7e6b-4e88-a4fb-2d984b3fa9b4-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-ff52b\" (UID: \"83cba701-7e6b-4e88-a4fb-2d984b3fa9b4\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" Jul 06 20:37:14.731015 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:14.730976 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" Jul 06 20:37:14.907272 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:14.907206 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sxdx7" Jul 06 20:37:15.259647 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:15.259569 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b96775d75-sw4cx"] Jul 06 20:37:16.488729 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:16.488626 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b"] Jul 06 20:37:16.490841 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:37:16.490804 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cba701_7e6b_4e88_a4fb_2d984b3fa9b4.slice/crio-97579cd406becf0d1629c9b1a5b6336df70560c9ca90182fc374523480b924cc WatchSource:0}: Error finding container 97579cd406becf0d1629c9b1a5b6336df70560c9ca90182fc374523480b924cc: Status 404 returned error can't find the container with id 97579cd406becf0d1629c9b1a5b6336df70560c9ca90182fc374523480b924cc Jul 06 20:37:16.543672 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:16.543565 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b96775d75-sw4cx"] Jul 06 20:37:16.546195 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:37:16.546155 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcd74e1_add1_4329_a05b_89be4d575b0d.slice/crio-3b1ac2f67f158cbedf4caca81bc77d446df45edf66fc6ed239a224777e3299c7 WatchSource:0}: Error finding container 3b1ac2f67f158cbedf4caca81bc77d446df45edf66fc6ed239a224777e3299c7: Status 404 returned error can't find the container with id 3b1ac2f67f158cbedf4caca81bc77d446df45edf66fc6ed239a224777e3299c7 Jul 06 20:37:16.710554 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:16.710517 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f6567c9b6-d6gt6"] Jul 06 20:37:16.713729 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:16.713693 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj"] Jul 06 20:37:16.714229 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:37:16.714166 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194f7868_6ebd_4220_a09d_d21f124c5c2d.slice/crio-48c5cc114758b2f7bb455ec23321403d724a57e493ddd2d3da3c2718a8108ad1 WatchSource:0}: Error finding container 48c5cc114758b2f7bb455ec23321403d724a57e493ddd2d3da3c2718a8108ad1: Status 404 returned error can't find the container with id 48c5cc114758b2f7bb455ec23321403d724a57e493ddd2d3da3c2718a8108ad1 Jul 06 20:37:16.716163 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:37:16.716141 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc366d3eb_8e1c_4934_9039_50bc149983eb.slice/crio-adcae83ca36a4d163f5c3b48a46b0082c3b73e87428d7b928d6438217b764914 WatchSource:0}: Error finding container adcae83ca36a4d163f5c3b48a46b0082c3b73e87428d7b928d6438217b764914: Status 404 returned error can't find the container with id adcae83ca36a4d163f5c3b48a46b0082c3b73e87428d7b928d6438217b764914 Jul 06 20:37:17.057840 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:17.057680 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" event={"ID":"194f7868-6ebd-4220-a09d-d21f124c5c2d","Type":"ContainerStarted","Data":"48c5cc114758b2f7bb455ec23321403d724a57e493ddd2d3da3c2718a8108ad1"} Jul 06 20:37:17.060255 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:17.060166 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bv7x" event={"ID":"91ad3522-11b2-4d4d-b6a5-d913f4b24933","Type":"ContainerStarted","Data":"4c60070ecaa356a8c68d3f5367a59ad8b568d7557a48ec603cae159f15d4d8b2"} Jul 06 20:37:17.062242 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:17.062162 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" event={"ID":"c366d3eb-8e1c-4934-9039-50bc149983eb","Type":"ContainerStarted","Data":"adcae83ca36a4d163f5c3b48a46b0082c3b73e87428d7b928d6438217b764914"} Jul 06 20:37:17.065037 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:17.064968 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" event={"ID":"83cba701-7e6b-4e88-a4fb-2d984b3fa9b4","Type":"ContainerStarted","Data":"97579cd406becf0d1629c9b1a5b6336df70560c9ca90182fc374523480b924cc"} Jul 06 20:37:17.068141 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:17.067710 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-gvh6p" event={"ID":"9f0b356c-3738-4e8e-a2f4-8b159924540a","Type":"ContainerStarted","Data":"59420a459934af919253fe29f223132e4e1057eeb85b9ec62748b9118dd178fd"} Jul 06 20:37:17.069501 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:17.069469 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-gvh6p" Jul 06 20:37:17.071984 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:17.071926 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" event={"ID":"0ffc2142-4f78-450c-bbfe-c7db04b949c6","Type":"ContainerStarted","Data":"c0248a72ef00eac40bae4c9123dce287062e328ff5227e8d74d75a94e5f75a33"} Jul 06 20:37:17.072684 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:17.072627 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" Jul 06 20:37:17.074049 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:17.073887 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b96775d75-sw4cx" event={"ID":"6bcd74e1-add1-4329-a05b-89be4d575b0d","Type":"ContainerStarted","Data":"3b1ac2f67f158cbedf4caca81bc77d446df45edf66fc6ed239a224777e3299c7"} Jul 06 20:37:17.075371 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:17.075327 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" Jul 06 20:37:17.080506 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:17.080471 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-gvh6p" Jul 06 20:37:17.099136 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:17.098476 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-gvh6p" podStartSLOduration=1.7945399869999998 podStartE2EDuration="23.098457479s" totalImagesPullingTime="21.303917492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:17.096164058 +0000 UTC m=+104.181719250" watchObservedRunningTime="2026-07-06 20:37:17.098457479 +0000 UTC m=+104.184012673" Jul 06 20:37:19.544002 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:19.543784 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-666c749768-mbfkv" podUID="dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" containerName="registry" containerID="cri-o://b03ec099393b71ad6745776f1d16a96289ad9a5f53b03aba28e68c6288a17b59" gracePeriod=30 Jul 06 20:37:22.108074 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.108043 2582 generic.go:356] "Generic (PLEG): container finished" podID="dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" containerID="b03ec099393b71ad6745776f1d16a96289ad9a5f53b03aba28e68c6288a17b59" exitCode=0 Jul 06 20:37:22.108439 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.108122 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666c749768-mbfkv" event={"ID":"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a","Type":"ContainerDied","Data":"b03ec099393b71ad6745776f1d16a96289ad9a5f53b03aba28e68c6288a17b59"} Jul 06 20:37:22.248537 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.248261 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:37:22.278143 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.277122 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-df86c5d5-h8v9j" podStartSLOduration=6.884920539 podStartE2EDuration="28.277103441s" totalImagesPullingTime="21.392182902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:17.14978246 +0000 UTC m=+104.235337653" watchObservedRunningTime="2026-07-06 20:37:22.277103441 +0000 UTC m=+109.362658636" Jul 06 20:37:22.398015 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.397987 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-ca-trust-extracted\") pod \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " Jul 06 20:37:22.398132 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.398021 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-trusted-ca\") pod \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " Jul 06 20:37:22.398132 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.398050 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-image-registry-private-configuration\") pod \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " Jul 06 20:37:22.398132 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.398073 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-certificates\") pod \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " Jul 06 20:37:22.398132 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.398096 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls\") pod \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " Jul 06 20:37:22.398132 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.398117 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-bound-sa-token\") pod \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " Jul 06 20:37:22.398370 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.398144 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-installation-pull-secrets\") pod \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " Jul 06 20:37:22.398370 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.398205 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qskqw\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-kube-api-access-qskqw\") pod \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\" (UID: \"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a\") " Jul 06 20:37:22.400114 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.398863 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" (UID: "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:37:22.400114 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.398995 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" (UID: "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:37:22.403972 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.402404 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-kube-api-access-qskqw" (OuterVolumeSpecName: "kube-api-access-qskqw") pod "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" (UID: "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a"). InnerVolumeSpecName "kube-api-access-qskqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:37:22.403972 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.402845 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" (UID: "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:37:22.403972 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.402969 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" (UID: "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:37:22.403972 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.403337 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" (UID: "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:37:22.406036 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.405585 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" (UID: "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:37:22.411670 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.411392 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" (UID: "dea9f7b1-5211-44d3-90c5-a4ec538c8e6a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:37:22.499779 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.499687 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qskqw\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-kube-api-access-qskqw\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:22.499779 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.499719 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-ca-trust-extracted\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:22.499779 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.499735 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-trusted-ca\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:22.499986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.499750 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-image-registry-private-configuration\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:22.499986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.499879 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-certificates\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:22.499986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.499894 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-registry-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:22.499986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.499911 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-bound-sa-token\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:22.499986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:22.499924 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a-installation-pull-secrets\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:23.115951 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.115914 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" event={"ID":"c366d3eb-8e1c-4934-9039-50bc149983eb","Type":"ContainerStarted","Data":"e5369d8be516cdb9158d5ef1f131d4817652b307e6c90505160aa1ebac14cf26"} Jul 06 20:37:23.116465 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.116013 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" event={"ID":"c366d3eb-8e1c-4934-9039-50bc149983eb","Type":"ContainerStarted","Data":"24b2530af3ba42ba34b1b3a4abe6664e316a63e7a4c3f3519087f832b3458b73"} Jul 06 20:37:23.116465 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.116039 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" event={"ID":"c366d3eb-8e1c-4934-9039-50bc149983eb","Type":"ContainerStarted","Data":"692f6b60c88f20e049a54fd292e81ed2ddeb2928f54875093331314d3121a7d8"} Jul 06 20:37:23.117726 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.117699 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666c749768-mbfkv" Jul 06 20:37:23.117726 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.117718 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666c749768-mbfkv" event={"ID":"dea9f7b1-5211-44d3-90c5-a4ec538c8e6a","Type":"ContainerDied","Data":"946ccb9f373d40c6e06a96b64a56c7e8c885fadca1b94cee6d1ceab379438ece"} Jul 06 20:37:23.117935 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.117794 2582 scope.go:117] "RemoveContainer" containerID="b03ec099393b71ad6745776f1d16a96289ad9a5f53b03aba28e68c6288a17b59" Jul 06 20:37:23.120202 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.120168 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" event={"ID":"83cba701-7e6b-4e88-a4fb-2d984b3fa9b4","Type":"ContainerStarted","Data":"95d49b5dbc748697930d0e4fe6ef8658f19df2b5770a1e393d4cffd2c56214c4"} Jul 06 20:37:23.120703 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.120658 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" Jul 06 20:37:23.123183 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.122292 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b96775d75-sw4cx" event={"ID":"6bcd74e1-add1-4329-a05b-89be4d575b0d","Type":"ContainerStarted","Data":"45b5c071b7e5c7fc53cbf0c87b718fce0c3d2f01cab24c5b73e1d350816e2c75"} Jul 06 20:37:23.127957 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.125124 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" event={"ID":"194f7868-6ebd-4220-a09d-d21f124c5c2d","Type":"ContainerStarted","Data":"37ad0f302da837b2197c717db9ec80d3e8cb51aad6db4233d02e27e48a0e4890"} Jul 06 20:37:23.127957 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.127859 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" Jul 06 20:37:23.131890 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.129808 2582 generic.go:356] "Generic (PLEG): container finished" podID="91ad3522-11b2-4d4d-b6a5-d913f4b24933" containerID="4751b503b1be1d98c67f3cbb235fef34ac3db1eb6e82ccdd014a7325c0cf84c4" exitCode=0 Jul 06 20:37:23.131890 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.129840 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bv7x" event={"ID":"91ad3522-11b2-4d4d-b6a5-d913f4b24933","Type":"ContainerDied","Data":"4751b503b1be1d98c67f3cbb235fef34ac3db1eb6e82ccdd014a7325c0cf84c4"} Jul 06 20:37:23.172452 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.172268 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-rxxbj" podStartSLOduration=9.81389593 podStartE2EDuration="15.172252384s" totalImagesPullingTime="5.358356454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:23.169489172 +0000 UTC m=+110.255044356" watchObservedRunningTime="2026-07-06 20:37:23.172252384 +0000 UTC m=+110.257807577" Jul 06 20:37:23.274633 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.272451 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-ff52b" podStartSLOduration=5.6925200799999995 podStartE2EDuration="11.272432853s" totalImagesPullingTime="5.579912773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:23.27052932 +0000 UTC m=+110.356084513" watchObservedRunningTime="2026-07-06 20:37:23.272432853 +0000 UTC m=+110.357988047" Jul 06 20:37:23.298928 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.298546 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666c749768-mbfkv"] Jul 06 20:37:23.298928 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.298832 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-666c749768-mbfkv"] Jul 06 20:37:23.320258 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.320213 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" podStartSLOduration=5.960201531 podStartE2EDuration="11.320168009s" totalImagesPullingTime="5.359966478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:23.317914498 +0000 UTC m=+110.403469703" watchObservedRunningTime="2026-07-06 20:37:23.320168009 +0000 UTC m=+110.405723200" Jul 06 20:37:23.376132 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.376087 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b96775d75-sw4cx" podStartSLOduration=9.839304027 podStartE2EDuration="15.376069857s" totalImagesPullingTime="5.53676583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:23.375383417 +0000 UTC m=+110.460938611" watchObservedRunningTime="2026-07-06 20:37:23.376069857 +0000 UTC m=+110.461625051" Jul 06 20:37:23.541733 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:23.541694 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" path="/var/lib/kubelet/pods/dea9f7b1-5211-44d3-90c5-a4ec538c8e6a/volumes" Jul 06 20:37:24.140232 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:24.140194 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bv7x" event={"ID":"91ad3522-11b2-4d4d-b6a5-d913f4b24933","Type":"ContainerStarted","Data":"7849d255dfa7c8b27cfff02458dcfa46855461ec652fb9c2a5e192a6fc0243b0"} Jul 06 20:37:24.140837 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:24.140257 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bv7x" event={"ID":"91ad3522-11b2-4d4d-b6a5-d913f4b24933","Type":"ContainerStarted","Data":"c2efbf80f3e132cacf75354be93ac2478f68a2d94039cf7775297efb135e4670"} Jul 06 20:37:24.163130 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:24.163075 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7bv7x" podStartSLOduration=10.312207729 podStartE2EDuration="16.163060691s" totalImagesPullingTime="5.725809775s" totalInitContainerRuntime="125.043187ms" isStatefulPod=true podCreationTimestamp="2026-07-06 20:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:24.161561852 +0000 UTC m=+111.247117046" watchObservedRunningTime="2026-07-06 20:37:24.163060691 +0000 UTC m=+111.248615883" Jul 06 20:37:29.174152 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:29.174119 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:33.017938 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:33.017905 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:33.018373 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:33.017961 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:42.245414 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:42.245385 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:37:42.247389 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:42.247361 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d976ba8-6cb4-4882-9f06-e9c1d62cc475-metrics-certs\") pod \"network-metrics-daemon-d26rt\" (UID: \"4d976ba8-6cb4-4882-9f06-e9c1d62cc475\") " pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:37:42.353102 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:42.353078 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sxc6n\"" Jul 06 20:37:42.360494 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:42.360477 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d26rt" Jul 06 20:37:42.513290 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:42.513221 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d26rt"] Jul 06 20:37:42.515809 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:37:42.515788 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d976ba8_6cb4_4882_9f06_e9c1d62cc475.slice/crio-36b6cd57a2616d53c42ab8e0a1b438d0df7323fa9297c44fef03f4a8ec6317eb WatchSource:0}: Error finding container 36b6cd57a2616d53c42ab8e0a1b438d0df7323fa9297c44fef03f4a8ec6317eb: Status 404 returned error can't find the container with id 36b6cd57a2616d53c42ab8e0a1b438d0df7323fa9297c44fef03f4a8ec6317eb Jul 06 20:37:43.209091 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:43.209049 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d26rt" event={"ID":"4d976ba8-6cb4-4882-9f06-e9c1d62cc475","Type":"ContainerStarted","Data":"36b6cd57a2616d53c42ab8e0a1b438d0df7323fa9297c44fef03f4a8ec6317eb"} Jul 06 20:37:44.214902 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:44.214869 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d26rt" event={"ID":"4d976ba8-6cb4-4882-9f06-e9c1d62cc475","Type":"ContainerStarted","Data":"723d1ff25b8d2aaed8fb5a13ee9d8bdd2c21ea91db9449885401ff6ad8f3e79e"} Jul 06 20:37:44.215215 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:44.214920 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d26rt" event={"ID":"4d976ba8-6cb4-4882-9f06-e9c1d62cc475","Type":"ContainerStarted","Data":"8c0c516cae37980085ab767ee7bd35faef3a279f28cddca2798faa578120fed2"} Jul 06 20:37:44.234894 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:44.234847 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d26rt" podStartSLOduration=128.833323139 podStartE2EDuration="2m10.234831546s" totalImagesPullingTime="1.401508407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:44.233489662 +0000 UTC m=+131.319044864" watchObservedRunningTime="2026-07-06 20:37:44.234831546 +0000 UTC m=+131.320386739" Jul 06 20:37:48.148064 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.148020 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7b96775d75-sw4cx" podUID="6bcd74e1-add1-4329-a05b-89be4d575b0d" containerName="console" containerID="cri-o://45b5c071b7e5c7fc53cbf0c87b718fce0c3d2f01cab24c5b73e1d350816e2c75" gracePeriod=15 Jul 06 20:37:48.427661 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.427634 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:48.593171 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.593141 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbbrh\" (UniqueName: \"kubernetes.io/projected/6bcd74e1-add1-4329-a05b-89be4d575b0d-kube-api-access-gbbrh\") pod \"6bcd74e1-add1-4329-a05b-89be4d575b0d\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " Jul 06 20:37:48.593171 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.593173 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-trusted-ca-bundle\") pod \"6bcd74e1-add1-4329-a05b-89be4d575b0d\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " Jul 06 20:37:48.593374 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.593195 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-oauth-serving-cert\") pod \"6bcd74e1-add1-4329-a05b-89be4d575b0d\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " Jul 06 20:37:48.593374 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.593208 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-service-ca\") pod \"6bcd74e1-add1-4329-a05b-89be4d575b0d\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " Jul 06 20:37:48.593374 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.593227 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-oauth-config\") pod \"6bcd74e1-add1-4329-a05b-89be4d575b0d\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " Jul 06 20:37:48.593374 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.593316 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-serving-cert\") pod \"6bcd74e1-add1-4329-a05b-89be4d575b0d\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " Jul 06 20:37:48.593374 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.593360 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-config\") pod \"6bcd74e1-add1-4329-a05b-89be4d575b0d\" (UID: \"6bcd74e1-add1-4329-a05b-89be4d575b0d\") " Jul 06 20:37:48.593713 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.593672 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-service-ca" (OuterVolumeSpecName: "service-ca") pod "6bcd74e1-add1-4329-a05b-89be4d575b0d" (UID: "6bcd74e1-add1-4329-a05b-89be4d575b0d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:37:48.593713 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.593695 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-config" (OuterVolumeSpecName: "console-config") pod "6bcd74e1-add1-4329-a05b-89be4d575b0d" (UID: "6bcd74e1-add1-4329-a05b-89be4d575b0d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:37:48.593869 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.593846 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6bcd74e1-add1-4329-a05b-89be4d575b0d" (UID: "6bcd74e1-add1-4329-a05b-89be4d575b0d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:37:48.593917 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.593859 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6bcd74e1-add1-4329-a05b-89be4d575b0d" (UID: "6bcd74e1-add1-4329-a05b-89be4d575b0d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:37:48.595375 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.595345 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6bcd74e1-add1-4329-a05b-89be4d575b0d" (UID: "6bcd74e1-add1-4329-a05b-89be4d575b0d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:37:48.595478 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.595392 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6bcd74e1-add1-4329-a05b-89be4d575b0d" (UID: "6bcd74e1-add1-4329-a05b-89be4d575b0d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:37:48.595478 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.595403 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcd74e1-add1-4329-a05b-89be4d575b0d-kube-api-access-gbbrh" (OuterVolumeSpecName: "kube-api-access-gbbrh") pod "6bcd74e1-add1-4329-a05b-89be4d575b0d" (UID: "6bcd74e1-add1-4329-a05b-89be4d575b0d"). InnerVolumeSpecName "kube-api-access-gbbrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:37:48.694290 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.694241 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbbrh\" (UniqueName: \"kubernetes.io/projected/6bcd74e1-add1-4329-a05b-89be4d575b0d-kube-api-access-gbbrh\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:48.694290 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.694262 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-trusted-ca-bundle\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:48.694290 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.694270 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-oauth-serving-cert\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:48.694290 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.694279 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-service-ca\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:48.694290 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.694287 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-oauth-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:48.694477 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.694294 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-serving-cert\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:48.694477 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:48.694303 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6bcd74e1-add1-4329-a05b-89be4d575b0d-console-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:37:49.236247 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:49.236213 2582 generic.go:356] "Generic (PLEG): container finished" podID="6bcd74e1-add1-4329-a05b-89be4d575b0d" containerID="45b5c071b7e5c7fc53cbf0c87b718fce0c3d2f01cab24c5b73e1d350816e2c75" exitCode=0 Jul 06 20:37:49.236614 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:49.236262 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b96775d75-sw4cx" Jul 06 20:37:49.236614 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:49.236298 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b96775d75-sw4cx" event={"ID":"6bcd74e1-add1-4329-a05b-89be4d575b0d","Type":"ContainerDied","Data":"45b5c071b7e5c7fc53cbf0c87b718fce0c3d2f01cab24c5b73e1d350816e2c75"} Jul 06 20:37:49.236614 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:49.236348 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b96775d75-sw4cx" event={"ID":"6bcd74e1-add1-4329-a05b-89be4d575b0d","Type":"ContainerDied","Data":"3b1ac2f67f158cbedf4caca81bc77d446df45edf66fc6ed239a224777e3299c7"} Jul 06 20:37:49.236614 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:49.236369 2582 scope.go:117] "RemoveContainer" containerID="45b5c071b7e5c7fc53cbf0c87b718fce0c3d2f01cab24c5b73e1d350816e2c75" Jul 06 20:37:49.244305 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:49.244290 2582 scope.go:117] "RemoveContainer" containerID="45b5c071b7e5c7fc53cbf0c87b718fce0c3d2f01cab24c5b73e1d350816e2c75" Jul 06 20:37:49.244600 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:37:49.244580 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b5c071b7e5c7fc53cbf0c87b718fce0c3d2f01cab24c5b73e1d350816e2c75\": container with ID starting with 45b5c071b7e5c7fc53cbf0c87b718fce0c3d2f01cab24c5b73e1d350816e2c75 not found: ID does not exist" containerID="45b5c071b7e5c7fc53cbf0c87b718fce0c3d2f01cab24c5b73e1d350816e2c75" Jul 06 20:37:49.244643 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:49.244607 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b5c071b7e5c7fc53cbf0c87b718fce0c3d2f01cab24c5b73e1d350816e2c75"} err="failed to get container status \"45b5c071b7e5c7fc53cbf0c87b718fce0c3d2f01cab24c5b73e1d350816e2c75\": rpc error: code = NotFound desc = could not find container \"45b5c071b7e5c7fc53cbf0c87b718fce0c3d2f01cab24c5b73e1d350816e2c75\": container with ID starting with 45b5c071b7e5c7fc53cbf0c87b718fce0c3d2f01cab24c5b73e1d350816e2c75 not found: ID does not exist" Jul 06 20:37:49.264083 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:49.264062 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b96775d75-sw4cx"] Jul 06 20:37:49.276542 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:49.276520 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b96775d75-sw4cx"] Jul 06 20:37:49.536494 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:49.536445 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bcd74e1-add1-4329-a05b-89be4d575b0d" path="/var/lib/kubelet/pods/6bcd74e1-add1-4329-a05b-89be4d575b0d/volumes" Jul 06 20:37:53.022596 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:53.022572 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:37:53.026200 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:37:53.026180 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f6567c9b6-d6gt6" Jul 06 20:38:36.206659 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.206627 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84cbbb6df6-djmdp"] Jul 06 20:38:36.207071 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.206911 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" containerName="registry" Jul 06 20:38:36.207071 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.206924 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" containerName="registry" Jul 06 20:38:36.207071 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.206942 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcd74e1-add1-4329-a05b-89be4d575b0d" containerName="console" Jul 06 20:38:36.207071 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.206948 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcd74e1-add1-4329-a05b-89be4d575b0d" containerName="console" Jul 06 20:38:36.207071 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.206988 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bcd74e1-add1-4329-a05b-89be4d575b0d" containerName="console" Jul 06 20:38:36.207071 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.206997 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dea9f7b1-5211-44d3-90c5-a4ec538c8e6a" containerName="registry" Jul 06 20:38:36.209428 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.209412 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.213079 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.213054 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 20:38:36.213079 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.213070 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 20:38:36.214424 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.214401 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 20:38:36.214549 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.214486 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qzqw5\"" Jul 06 20:38:36.216120 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.215973 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 20:38:36.216195 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.216180 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 20:38:36.224978 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.224960 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84cbbb6df6-djmdp"] Jul 06 20:38:36.233453 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.233437 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 20:38:36.305545 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.305521 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-console-config\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.305637 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.305553 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74159a70-6cff-4925-8e77-827be08df289-console-oauth-config\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.305637 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.305572 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74159a70-6cff-4925-8e77-827be08df289-console-serving-cert\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.305637 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.305594 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksgl4\" (UniqueName: \"kubernetes.io/projected/74159a70-6cff-4925-8e77-827be08df289-kube-api-access-ksgl4\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.305637 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.305609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-trusted-ca-bundle\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.305637 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.305629 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-oauth-serving-cert\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.305852 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.305690 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-service-ca\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.406285 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.406261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-service-ca\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.406380 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.406307 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-console-config\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.406380 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.406341 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74159a70-6cff-4925-8e77-827be08df289-console-oauth-config\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.406380 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.406365 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74159a70-6cff-4925-8e77-827be08df289-console-serving-cert\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.406533 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.406403 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksgl4\" (UniqueName: \"kubernetes.io/projected/74159a70-6cff-4925-8e77-827be08df289-kube-api-access-ksgl4\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.406533 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.406427 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-trusted-ca-bundle\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.406533 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.406458 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-oauth-serving-cert\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.407015 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.406993 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-service-ca\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.407145 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.407069 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-console-config\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.407211 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.407174 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-oauth-serving-cert\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.407334 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.407312 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-trusted-ca-bundle\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.408751 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.408735 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74159a70-6cff-4925-8e77-827be08df289-console-oauth-config\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.408820 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.408803 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74159a70-6cff-4925-8e77-827be08df289-console-serving-cert\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.416719 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.416702 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksgl4\" (UniqueName: \"kubernetes.io/projected/74159a70-6cff-4925-8e77-827be08df289-kube-api-access-ksgl4\") pod \"console-84cbbb6df6-djmdp\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.519098 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.519052 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:36.645997 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:36.645971 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84cbbb6df6-djmdp"] Jul 06 20:38:36.648090 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:38:36.648068 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74159a70_6cff_4925_8e77_827be08df289.slice/crio-74d5b69e9c8cb8cc507a824dd4d3c2caa2588f18cb0c5f7ce956e21ee4cee732 WatchSource:0}: Error finding container 74d5b69e9c8cb8cc507a824dd4d3c2caa2588f18cb0c5f7ce956e21ee4cee732: Status 404 returned error can't find the container with id 74d5b69e9c8cb8cc507a824dd4d3c2caa2588f18cb0c5f7ce956e21ee4cee732 Jul 06 20:38:37.408394 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:37.408359 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cbbb6df6-djmdp" event={"ID":"74159a70-6cff-4925-8e77-827be08df289","Type":"ContainerStarted","Data":"09f557bf63a0664011156a0f6d529292fc2d8be87dec37a5cea87b813fab5190"} Jul 06 20:38:37.408747 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:37.408448 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cbbb6df6-djmdp" event={"ID":"74159a70-6cff-4925-8e77-827be08df289","Type":"ContainerStarted","Data":"74d5b69e9c8cb8cc507a824dd4d3c2caa2588f18cb0c5f7ce956e21ee4cee732"} Jul 06 20:38:37.436920 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:37.436877 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84cbbb6df6-djmdp" podStartSLOduration=1.436862875 podStartE2EDuration="1.436862875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:38:37.436710644 +0000 UTC m=+184.522265834" watchObservedRunningTime="2026-07-06 20:38:37.436862875 +0000 UTC m=+184.522418068" Jul 06 20:38:44.225726 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.225688 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84cbbb6df6-djmdp"] Jul 06 20:38:44.261497 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.261468 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54d8449695-59jzs"] Jul 06 20:38:44.263745 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.263731 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.276699 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.276674 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d8449695-59jzs"] Jul 06 20:38:44.357265 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.357246 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-config\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.357368 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.357277 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-oauth-serving-cert\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.357368 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.357301 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-oauth-config\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.357368 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.357363 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-serving-cert\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.357462 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.357393 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqmqh\" (UniqueName: \"kubernetes.io/projected/35231930-75b0-4f52-bf5c-fee7cc8990c9-kube-api-access-sqmqh\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.357462 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.357428 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-service-ca\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.357462 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.357442 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-trusted-ca-bundle\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.458417 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.458397 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-oauth-serving-cert\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.458522 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.458424 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-oauth-config\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.458522 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.458441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-serving-cert\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.458522 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.458464 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqmqh\" (UniqueName: \"kubernetes.io/projected/35231930-75b0-4f52-bf5c-fee7cc8990c9-kube-api-access-sqmqh\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.458522 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.458489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-service-ca\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.458522 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.458504 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-trusted-ca-bundle\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.458807 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.458530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-config\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.459274 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.459250 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-oauth-serving-cert\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.459383 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.459255 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-config\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.459383 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.459291 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-service-ca\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.459506 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.459481 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-trusted-ca-bundle\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.460845 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.460819 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-serving-cert\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.460943 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.460858 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-oauth-config\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.468227 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.468204 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqmqh\" (UniqueName: \"kubernetes.io/projected/35231930-75b0-4f52-bf5c-fee7cc8990c9-kube-api-access-sqmqh\") pod \"console-54d8449695-59jzs\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.573407 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.573351 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:44.703495 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:44.703464 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d8449695-59jzs"] Jul 06 20:38:44.707043 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:38:44.707015 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35231930_75b0_4f52_bf5c_fee7cc8990c9.slice/crio-a5a3e37dffdaf7df3ce4370a7b04f57ecba7d1aedb997c52184d369d3b745c4b WatchSource:0}: Error finding container a5a3e37dffdaf7df3ce4370a7b04f57ecba7d1aedb997c52184d369d3b745c4b: Status 404 returned error can't find the container with id a5a3e37dffdaf7df3ce4370a7b04f57ecba7d1aedb997c52184d369d3b745c4b Jul 06 20:38:45.437744 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:45.437708 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d8449695-59jzs" event={"ID":"35231930-75b0-4f52-bf5c-fee7cc8990c9","Type":"ContainerStarted","Data":"ce727af982b11993554e241878a78fdaa2df84255f09a8f87f778bd568a287c6"} Jul 06 20:38:45.438106 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:45.437773 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d8449695-59jzs" event={"ID":"35231930-75b0-4f52-bf5c-fee7cc8990c9","Type":"ContainerStarted","Data":"a5a3e37dffdaf7df3ce4370a7b04f57ecba7d1aedb997c52184d369d3b745c4b"} Jul 06 20:38:45.461101 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:45.461052 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54d8449695-59jzs" podStartSLOduration=1.461034 podStartE2EDuration="1.461034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:38:45.460145571 +0000 UTC m=+192.545700764" watchObservedRunningTime="2026-07-06 20:38:45.461034 +0000 UTC m=+192.546589192" Jul 06 20:38:46.519850 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:46.519811 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:38:54.574433 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:54.574403 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:54.574979 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:54.574457 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:54.578824 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:54.578802 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:38:55.478839 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:38:55.478817 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:39:09.244480 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.244431 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-84cbbb6df6-djmdp" podUID="74159a70-6cff-4925-8e77-827be08df289" containerName="console" containerID="cri-o://09f557bf63a0664011156a0f6d529292fc2d8be87dec37a5cea87b813fab5190" gracePeriod=15 Jul 06 20:39:09.476157 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.476136 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:39:09.525445 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.525377 2582 generic.go:356] "Generic (PLEG): container finished" podID="74159a70-6cff-4925-8e77-827be08df289" containerID="09f557bf63a0664011156a0f6d529292fc2d8be87dec37a5cea87b813fab5190" exitCode=0 Jul 06 20:39:09.525445 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.525420 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cbbb6df6-djmdp" Jul 06 20:39:09.525573 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.525452 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cbbb6df6-djmdp" event={"ID":"74159a70-6cff-4925-8e77-827be08df289","Type":"ContainerDied","Data":"09f557bf63a0664011156a0f6d529292fc2d8be87dec37a5cea87b813fab5190"} Jul 06 20:39:09.525573 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.525502 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cbbb6df6-djmdp" event={"ID":"74159a70-6cff-4925-8e77-827be08df289","Type":"ContainerDied","Data":"74d5b69e9c8cb8cc507a824dd4d3c2caa2588f18cb0c5f7ce956e21ee4cee732"} Jul 06 20:39:09.525573 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.525529 2582 scope.go:117] "RemoveContainer" containerID="09f557bf63a0664011156a0f6d529292fc2d8be87dec37a5cea87b813fab5190" Jul 06 20:39:09.529808 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.529787 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-console-config\") pod \"74159a70-6cff-4925-8e77-827be08df289\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " Jul 06 20:39:09.529929 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.529825 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74159a70-6cff-4925-8e77-827be08df289-console-oauth-config\") pod \"74159a70-6cff-4925-8e77-827be08df289\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " Jul 06 20:39:09.529929 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.529868 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-oauth-serving-cert\") pod \"74159a70-6cff-4925-8e77-827be08df289\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " Jul 06 20:39:09.530044 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.529946 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksgl4\" (UniqueName: \"kubernetes.io/projected/74159a70-6cff-4925-8e77-827be08df289-kube-api-access-ksgl4\") pod \"74159a70-6cff-4925-8e77-827be08df289\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " Jul 06 20:39:09.530044 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.529967 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-trusted-ca-bundle\") pod \"74159a70-6cff-4925-8e77-827be08df289\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " Jul 06 20:39:09.530044 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.530009 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-service-ca\") pod \"74159a70-6cff-4925-8e77-827be08df289\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " Jul 06 20:39:09.530044 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.530036 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74159a70-6cff-4925-8e77-827be08df289-console-serving-cert\") pod \"74159a70-6cff-4925-8e77-827be08df289\" (UID: \"74159a70-6cff-4925-8e77-827be08df289\") " Jul 06 20:39:09.530348 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.530323 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "74159a70-6cff-4925-8e77-827be08df289" (UID: "74159a70-6cff-4925-8e77-827be08df289"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:39:09.530348 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.530335 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-console-config" (OuterVolumeSpecName: "console-config") pod "74159a70-6cff-4925-8e77-827be08df289" (UID: "74159a70-6cff-4925-8e77-827be08df289"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:39:09.530507 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.530406 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "74159a70-6cff-4925-8e77-827be08df289" (UID: "74159a70-6cff-4925-8e77-827be08df289"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:39:09.530657 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.530635 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-service-ca" (OuterVolumeSpecName: "service-ca") pod "74159a70-6cff-4925-8e77-827be08df289" (UID: "74159a70-6cff-4925-8e77-827be08df289"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:39:09.532241 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.532211 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74159a70-6cff-4925-8e77-827be08df289-kube-api-access-ksgl4" (OuterVolumeSpecName: "kube-api-access-ksgl4") pod "74159a70-6cff-4925-8e77-827be08df289" (UID: "74159a70-6cff-4925-8e77-827be08df289"). InnerVolumeSpecName "kube-api-access-ksgl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:39:09.532385 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.532361 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74159a70-6cff-4925-8e77-827be08df289-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "74159a70-6cff-4925-8e77-827be08df289" (UID: "74159a70-6cff-4925-8e77-827be08df289"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:39:09.532469 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.532448 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74159a70-6cff-4925-8e77-827be08df289-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "74159a70-6cff-4925-8e77-827be08df289" (UID: "74159a70-6cff-4925-8e77-827be08df289"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:39:09.534288 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.534271 2582 scope.go:117] "RemoveContainer" containerID="09f557bf63a0664011156a0f6d529292fc2d8be87dec37a5cea87b813fab5190" Jul 06 20:39:09.534572 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:39:09.534543 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f557bf63a0664011156a0f6d529292fc2d8be87dec37a5cea87b813fab5190\": container with ID starting with 09f557bf63a0664011156a0f6d529292fc2d8be87dec37a5cea87b813fab5190 not found: ID does not exist" containerID="09f557bf63a0664011156a0f6d529292fc2d8be87dec37a5cea87b813fab5190" Jul 06 20:39:09.534661 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.534580 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f557bf63a0664011156a0f6d529292fc2d8be87dec37a5cea87b813fab5190"} err="failed to get container status \"09f557bf63a0664011156a0f6d529292fc2d8be87dec37a5cea87b813fab5190\": rpc error: code = NotFound desc = could not find container \"09f557bf63a0664011156a0f6d529292fc2d8be87dec37a5cea87b813fab5190\": container with ID starting with 09f557bf63a0664011156a0f6d529292fc2d8be87dec37a5cea87b813fab5190 not found: ID does not exist" Jul 06 20:39:09.630837 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.630814 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-oauth-serving-cert\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:39:09.630837 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.630834 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksgl4\" (UniqueName: \"kubernetes.io/projected/74159a70-6cff-4925-8e77-827be08df289-kube-api-access-ksgl4\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:39:09.630950 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.630843 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-trusted-ca-bundle\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:39:09.630950 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.630851 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-service-ca\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:39:09.630950 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.630860 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74159a70-6cff-4925-8e77-827be08df289-console-serving-cert\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:39:09.630950 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.630868 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74159a70-6cff-4925-8e77-827be08df289-console-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:39:09.630950 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.630875 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74159a70-6cff-4925-8e77-827be08df289-console-oauth-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:39:09.846614 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.846569 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84cbbb6df6-djmdp"] Jul 06 20:39:09.851351 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:09.851317 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84cbbb6df6-djmdp"] Jul 06 20:39:11.536004 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:39:11.535978 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74159a70-6cff-4925-8e77-827be08df289" path="/var/lib/kubelet/pods/74159a70-6cff-4925-8e77-827be08df289/volumes" Jul 06 20:40:15.023685 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.023645 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx"] Jul 06 20:40:15.024101 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.023958 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74159a70-6cff-4925-8e77-827be08df289" containerName="console" Jul 06 20:40:15.024101 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.023972 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="74159a70-6cff-4925-8e77-827be08df289" containerName="console" Jul 06 20:40:15.024101 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.024025 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="74159a70-6cff-4925-8e77-827be08df289" containerName="console" Jul 06 20:40:15.026017 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.025996 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" Jul 06 20:40:15.028749 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.028727 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 20:40:15.028863 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.028751 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 20:40:15.029844 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.029829 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7s2fn\"" Jul 06 20:40:15.036774 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.036745 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx"] Jul 06 20:40:15.156011 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.155992 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7afb8f-ba6b-4c96-ab80-39cde915876f-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx\" (UID: \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" Jul 06 20:40:15.156107 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.156024 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6j74\" (UniqueName: \"kubernetes.io/projected/cc7afb8f-ba6b-4c96-ab80-39cde915876f-kube-api-access-c6j74\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx\" (UID: \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" Jul 06 20:40:15.156107 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.156054 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7afb8f-ba6b-4c96-ab80-39cde915876f-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx\" (UID: \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" Jul 06 20:40:15.256673 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.256654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7afb8f-ba6b-4c96-ab80-39cde915876f-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx\" (UID: \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" Jul 06 20:40:15.256780 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.256686 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6j74\" (UniqueName: \"kubernetes.io/projected/cc7afb8f-ba6b-4c96-ab80-39cde915876f-kube-api-access-c6j74\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx\" (UID: \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" Jul 06 20:40:15.256780 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.256717 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7afb8f-ba6b-4c96-ab80-39cde915876f-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx\" (UID: \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" Jul 06 20:40:15.257000 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.256984 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7afb8f-ba6b-4c96-ab80-39cde915876f-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx\" (UID: \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" Jul 06 20:40:15.257067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.257051 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7afb8f-ba6b-4c96-ab80-39cde915876f-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx\" (UID: \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" Jul 06 20:40:15.267594 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.267567 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6j74\" (UniqueName: \"kubernetes.io/projected/cc7afb8f-ba6b-4c96-ab80-39cde915876f-kube-api-access-c6j74\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx\" (UID: \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" Jul 06 20:40:15.334426 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.334376 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" Jul 06 20:40:15.458605 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.458579 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx"] Jul 06 20:40:15.461437 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:40:15.461407 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7afb8f_ba6b_4c96_ab80_39cde915876f.slice/crio-4275a7c0b2c0e0a0bdb1222dc3379b92366964fb65c7d11a948b6b795ad531ca WatchSource:0}: Error finding container 4275a7c0b2c0e0a0bdb1222dc3379b92366964fb65c7d11a948b6b795ad531ca: Status 404 returned error can't find the container with id 4275a7c0b2c0e0a0bdb1222dc3379b92366964fb65c7d11a948b6b795ad531ca Jul 06 20:40:15.738081 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:15.738051 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" event={"ID":"cc7afb8f-ba6b-4c96-ab80-39cde915876f","Type":"ContainerStarted","Data":"4275a7c0b2c0e0a0bdb1222dc3379b92366964fb65c7d11a948b6b795ad531ca"} Jul 06 20:40:26.783625 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:26.783596 2582 generic.go:356] "Generic (PLEG): container finished" podID="cc7afb8f-ba6b-4c96-ab80-39cde915876f" containerID="0efabe0b9fced61580976bec644ce5efe037d44681ed6bed5634673c9c99c596" exitCode=0 Jul 06 20:40:26.783954 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:26.783653 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" event={"ID":"cc7afb8f-ba6b-4c96-ab80-39cde915876f","Type":"ContainerDied","Data":"0efabe0b9fced61580976bec644ce5efe037d44681ed6bed5634673c9c99c596"} Jul 06 20:40:29.797205 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:29.797175 2582 generic.go:356] "Generic (PLEG): container finished" podID="cc7afb8f-ba6b-4c96-ab80-39cde915876f" containerID="3ca257a7e24f9b7567b6696cb49325f1493b82a800508e3c2c5ccec9c24d3feb" exitCode=0 Jul 06 20:40:29.797523 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:29.797268 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" event={"ID":"cc7afb8f-ba6b-4c96-ab80-39cde915876f","Type":"ContainerDied","Data":"3ca257a7e24f9b7567b6696cb49325f1493b82a800508e3c2c5ccec9c24d3feb"} Jul 06 20:40:33.335308 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:33.335273 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 20:40:33.336375 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:33.336353 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 20:40:33.342505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:33.342485 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 20:40:38.833512 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:38.833489 2582 generic.go:356] "Generic (PLEG): container finished" podID="cc7afb8f-ba6b-4c96-ab80-39cde915876f" containerID="ac032d843dba58cf2e06a23340b74405572fd156099b0ebf7921a22ed746b791" exitCode=0 Jul 06 20:40:38.833851 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:38.833533 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" event={"ID":"cc7afb8f-ba6b-4c96-ab80-39cde915876f","Type":"ContainerDied","Data":"ac032d843dba58cf2e06a23340b74405572fd156099b0ebf7921a22ed746b791"} Jul 06 20:40:39.959663 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:39.959642 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" Jul 06 20:40:40.031375 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:40.026710 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7afb8f-ba6b-4c96-ab80-39cde915876f-bundle\") pod \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\" (UID: \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\") " Jul 06 20:40:40.031375 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:40.026793 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7afb8f-ba6b-4c96-ab80-39cde915876f-util\") pod \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\" (UID: \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\") " Jul 06 20:40:40.031375 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:40.026861 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6j74\" (UniqueName: \"kubernetes.io/projected/cc7afb8f-ba6b-4c96-ab80-39cde915876f-kube-api-access-c6j74\") pod \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\" (UID: \"cc7afb8f-ba6b-4c96-ab80-39cde915876f\") " Jul 06 20:40:40.031375 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:40.028204 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7afb8f-ba6b-4c96-ab80-39cde915876f-bundle" (OuterVolumeSpecName: "bundle") pod "cc7afb8f-ba6b-4c96-ab80-39cde915876f" (UID: "cc7afb8f-ba6b-4c96-ab80-39cde915876f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:40:40.033581 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:40.033554 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7afb8f-ba6b-4c96-ab80-39cde915876f-kube-api-access-c6j74" (OuterVolumeSpecName: "kube-api-access-c6j74") pod "cc7afb8f-ba6b-4c96-ab80-39cde915876f" (UID: "cc7afb8f-ba6b-4c96-ab80-39cde915876f"). InnerVolumeSpecName "kube-api-access-c6j74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:40:40.033923 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:40.033904 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7afb8f-ba6b-4c96-ab80-39cde915876f-util" (OuterVolumeSpecName: "util") pod "cc7afb8f-ba6b-4c96-ab80-39cde915876f" (UID: "cc7afb8f-ba6b-4c96-ab80-39cde915876f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:40:40.128376 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:40.128353 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6j74\" (UniqueName: \"kubernetes.io/projected/cc7afb8f-ba6b-4c96-ab80-39cde915876f-kube-api-access-c6j74\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:40:40.128376 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:40.128375 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7afb8f-ba6b-4c96-ab80-39cde915876f-bundle\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:40:40.128494 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:40.128384 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7afb8f-ba6b-4c96-ab80-39cde915876f-util\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:40:40.842042 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:40.842009 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" event={"ID":"cc7afb8f-ba6b-4c96-ab80-39cde915876f","Type":"ContainerDied","Data":"4275a7c0b2c0e0a0bdb1222dc3379b92366964fb65c7d11a948b6b795ad531ca"} Jul 06 20:40:40.842143 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:40.842056 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4275a7c0b2c0e0a0bdb1222dc3379b92366964fb65c7d11a948b6b795ad531ca" Jul 06 20:40:40.842143 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:40.842027 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589xqqpx" Jul 06 20:40:48.280330 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.280295 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h"] Jul 06 20:40:48.280715 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.280684 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7afb8f-ba6b-4c96-ab80-39cde915876f" containerName="extract" Jul 06 20:40:48.280715 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.280705 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7afb8f-ba6b-4c96-ab80-39cde915876f" containerName="extract" Jul 06 20:40:48.280808 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.280746 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7afb8f-ba6b-4c96-ab80-39cde915876f" containerName="util" Jul 06 20:40:48.280808 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.280773 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7afb8f-ba6b-4c96-ab80-39cde915876f" containerName="util" Jul 06 20:40:48.280808 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.280790 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7afb8f-ba6b-4c96-ab80-39cde915876f" containerName="pull" Jul 06 20:40:48.280808 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.280797 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7afb8f-ba6b-4c96-ab80-39cde915876f" containerName="pull" Jul 06 20:40:48.280932 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.280889 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7afb8f-ba6b-4c96-ab80-39cde915876f" containerName="extract" Jul 06 20:40:48.283688 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.283671 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h" Jul 06 20:40:48.288379 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.288357 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 06 20:40:48.288544 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.288531 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-xf8ww\"" Jul 06 20:40:48.289058 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.289043 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 06 20:40:48.289122 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.289108 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 06 20:40:48.299687 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.299664 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h"] Jul 06 20:40:48.387719 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.387695 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s54wl\" (UniqueName: \"kubernetes.io/projected/470d0499-1330-4f07-9a02-2c127027dd56-kube-api-access-s54wl\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h\" (UID: \"470d0499-1330-4f07-9a02-2c127027dd56\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h" Jul 06 20:40:48.387824 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.387725 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/470d0499-1330-4f07-9a02-2c127027dd56-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h\" (UID: \"470d0499-1330-4f07-9a02-2c127027dd56\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h" Jul 06 20:40:48.488463 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.488439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s54wl\" (UniqueName: \"kubernetes.io/projected/470d0499-1330-4f07-9a02-2c127027dd56-kube-api-access-s54wl\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h\" (UID: \"470d0499-1330-4f07-9a02-2c127027dd56\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h" Jul 06 20:40:48.488538 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.488473 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/470d0499-1330-4f07-9a02-2c127027dd56-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h\" (UID: \"470d0499-1330-4f07-9a02-2c127027dd56\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h" Jul 06 20:40:48.490589 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.490565 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/470d0499-1330-4f07-9a02-2c127027dd56-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h\" (UID: \"470d0499-1330-4f07-9a02-2c127027dd56\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h" Jul 06 20:40:48.498431 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.498410 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s54wl\" (UniqueName: \"kubernetes.io/projected/470d0499-1330-4f07-9a02-2c127027dd56-kube-api-access-s54wl\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h\" (UID: \"470d0499-1330-4f07-9a02-2c127027dd56\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h" Jul 06 20:40:48.595229 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.595180 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h" Jul 06 20:40:48.723225 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.723201 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h"] Jul 06 20:40:48.725197 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:40:48.725171 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470d0499_1330_4f07_9a02_2c127027dd56.slice/crio-bb817647a349ce16c38ee6f763d1e3adf1ac966e608ce0f49171b508af7efe8c WatchSource:0}: Error finding container bb817647a349ce16c38ee6f763d1e3adf1ac966e608ce0f49171b508af7efe8c: Status 404 returned error can't find the container with id bb817647a349ce16c38ee6f763d1e3adf1ac966e608ce0f49171b508af7efe8c Jul 06 20:40:48.726990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.726973 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:40:48.872821 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:48.872797 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h" event={"ID":"470d0499-1330-4f07-9a02-2c127027dd56","Type":"ContainerStarted","Data":"bb817647a349ce16c38ee6f763d1e3adf1ac966e608ce0f49171b508af7efe8c"} Jul 06 20:40:51.892351 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:51.892314 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h" event={"ID":"470d0499-1330-4f07-9a02-2c127027dd56","Type":"ContainerStarted","Data":"d49c80131185e651835cb2b9b4d27e8b8ce3aa1ff2f32ae22b599a44b1386c6c"} Jul 06 20:40:51.892744 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:51.892459 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h" Jul 06 20:40:51.923361 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:51.923306 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h" podStartSLOduration=1.278555737 podStartE2EDuration="3.923292318s" totalImagesPullingTime="2.644736581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:40:51.921828367 +0000 UTC m=+319.007383559" watchObservedRunningTime="2026-07-06 20:40:51.923292318 +0000 UTC m=+319.008847513" Jul 06 20:40:52.087911 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.087885 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-x7lck"] Jul 06 20:40:52.089886 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.089868 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:40:52.094356 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.094327 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Jul 06 20:40:52.094454 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.094360 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 06 20:40:52.094641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.094625 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-gxh54\"" Jul 06 20:40:52.104438 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.104418 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-x7lck"] Jul 06 20:40:52.113147 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.113119 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1aac1acb-f216-4365-8ee3-6e893c3904d4-certificates\") pod \"keda-operator-68bff59c-x7lck\" (UID: \"1aac1acb-f216-4365-8ee3-6e893c3904d4\") " pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:40:52.113229 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.113160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xmg\" (UniqueName: \"kubernetes.io/projected/1aac1acb-f216-4365-8ee3-6e893c3904d4-kube-api-access-j4xmg\") pod \"keda-operator-68bff59c-x7lck\" (UID: \"1aac1acb-f216-4365-8ee3-6e893c3904d4\") " pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:40:52.113229 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.113188 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1aac1acb-f216-4365-8ee3-6e893c3904d4-cabundle0\") pod \"keda-operator-68bff59c-x7lck\" (UID: \"1aac1acb-f216-4365-8ee3-6e893c3904d4\") " pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:40:52.213570 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.213508 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xmg\" (UniqueName: \"kubernetes.io/projected/1aac1acb-f216-4365-8ee3-6e893c3904d4-kube-api-access-j4xmg\") pod \"keda-operator-68bff59c-x7lck\" (UID: \"1aac1acb-f216-4365-8ee3-6e893c3904d4\") " pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:40:52.213570 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.213548 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1aac1acb-f216-4365-8ee3-6e893c3904d4-cabundle0\") pod \"keda-operator-68bff59c-x7lck\" (UID: \"1aac1acb-f216-4365-8ee3-6e893c3904d4\") " pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:40:52.213722 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.213620 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1aac1acb-f216-4365-8ee3-6e893c3904d4-certificates\") pod \"keda-operator-68bff59c-x7lck\" (UID: \"1aac1acb-f216-4365-8ee3-6e893c3904d4\") " pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:40:52.213804 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:52.213742 2582 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found Jul 06 20:40:52.213804 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:52.213788 2582 secret.go:281] references non-existent secret key: ca.crt Jul 06 20:40:52.213804 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:52.213800 2582 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 06 20:40:52.213945 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:52.213815 2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-x7lck: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Jul 06 20:40:52.213945 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:52.213887 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1aac1acb-f216-4365-8ee3-6e893c3904d4-certificates podName:1aac1acb-f216-4365-8ee3-6e893c3904d4 nodeName:}" failed. No retries permitted until 2026-07-06 20:40:52.713868693 +0000 UTC m=+319.799423866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1aac1acb-f216-4365-8ee3-6e893c3904d4-certificates") pod "keda-operator-68bff59c-x7lck" (UID: "1aac1acb-f216-4365-8ee3-6e893c3904d4") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Jul 06 20:40:52.214146 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.214128 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1aac1acb-f216-4365-8ee3-6e893c3904d4-cabundle0\") pod \"keda-operator-68bff59c-x7lck\" (UID: \"1aac1acb-f216-4365-8ee3-6e893c3904d4\") " pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:40:52.226030 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.226006 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xmg\" (UniqueName: \"kubernetes.io/projected/1aac1acb-f216-4365-8ee3-6e893c3904d4-kube-api-access-j4xmg\") pod \"keda-operator-68bff59c-x7lck\" (UID: \"1aac1acb-f216-4365-8ee3-6e893c3904d4\") " pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:40:52.326599 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.326574 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk"] Jul 06 20:40:52.328985 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.328968 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:40:52.331749 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.331733 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Jul 06 20:40:52.342531 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.342508 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk"] Jul 06 20:40:52.415286 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.415266 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/31ae1ae7-371b-4b6c-b85a-49447ad4fbd2-certificates\") pod \"keda-metrics-apiserver-559f75f947-p9jkk\" (UID: \"31ae1ae7-371b-4b6c-b85a-49447ad4fbd2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:40:52.415356 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.415302 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrz5d\" (UniqueName: \"kubernetes.io/projected/31ae1ae7-371b-4b6c-b85a-49447ad4fbd2-kube-api-access-vrz5d\") pod \"keda-metrics-apiserver-559f75f947-p9jkk\" (UID: \"31ae1ae7-371b-4b6c-b85a-49447ad4fbd2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:40:52.415356 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.415349 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/31ae1ae7-371b-4b6c-b85a-49447ad4fbd2-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-p9jkk\" (UID: \"31ae1ae7-371b-4b6c-b85a-49447ad4fbd2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:40:52.516526 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.516475 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/31ae1ae7-371b-4b6c-b85a-49447ad4fbd2-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-p9jkk\" (UID: \"31ae1ae7-371b-4b6c-b85a-49447ad4fbd2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:40:52.516526 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.516514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/31ae1ae7-371b-4b6c-b85a-49447ad4fbd2-certificates\") pod \"keda-metrics-apiserver-559f75f947-p9jkk\" (UID: \"31ae1ae7-371b-4b6c-b85a-49447ad4fbd2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:40:52.516637 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.516549 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrz5d\" (UniqueName: \"kubernetes.io/projected/31ae1ae7-371b-4b6c-b85a-49447ad4fbd2-kube-api-access-vrz5d\") pod \"keda-metrics-apiserver-559f75f947-p9jkk\" (UID: \"31ae1ae7-371b-4b6c-b85a-49447ad4fbd2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:40:52.516637 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:52.516620 2582 secret.go:281] references non-existent secret key: tls.crt Jul 06 20:40:52.516637 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:52.516636 2582 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt Jul 06 20:40:52.516740 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:52.516651 2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk: references non-existent secret key: tls.crt Jul 06 20:40:52.516740 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:52.516695 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31ae1ae7-371b-4b6c-b85a-49447ad4fbd2-certificates podName:31ae1ae7-371b-4b6c-b85a-49447ad4fbd2 nodeName:}" failed. No retries permitted until 2026-07-06 20:40:53.016682492 +0000 UTC m=+320.102237664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/31ae1ae7-371b-4b6c-b85a-49447ad4fbd2-certificates") pod "keda-metrics-apiserver-559f75f947-p9jkk" (UID: "31ae1ae7-371b-4b6c-b85a-49447ad4fbd2") : references non-existent secret key: tls.crt Jul 06 20:40:52.516907 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.516883 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/31ae1ae7-371b-4b6c-b85a-49447ad4fbd2-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-p9jkk\" (UID: \"31ae1ae7-371b-4b6c-b85a-49447ad4fbd2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:40:52.526449 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.526429 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrz5d\" (UniqueName: \"kubernetes.io/projected/31ae1ae7-371b-4b6c-b85a-49447ad4fbd2-kube-api-access-vrz5d\") pod \"keda-metrics-apiserver-559f75f947-p9jkk\" (UID: \"31ae1ae7-371b-4b6c-b85a-49447ad4fbd2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:40:52.718198 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.718172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1aac1acb-f216-4365-8ee3-6e893c3904d4-certificates\") pod \"keda-operator-68bff59c-x7lck\" (UID: \"1aac1acb-f216-4365-8ee3-6e893c3904d4\") " pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:40:52.720382 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.720358 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1aac1acb-f216-4365-8ee3-6e893c3904d4-certificates\") pod \"keda-operator-68bff59c-x7lck\" (UID: \"1aac1acb-f216-4365-8ee3-6e893c3904d4\") " pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:40:52.750639 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.750617 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-jj5rv"] Jul 06 20:40:52.753035 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.753022 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" Jul 06 20:40:52.755722 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.755704 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Jul 06 20:40:52.769805 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.769734 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-jj5rv"] Jul 06 20:40:52.819357 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.819335 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351-certificates\") pod \"keda-admission-5fbd5c4644-jj5rv\" (UID: \"3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351\") " pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" Jul 06 20:40:52.819493 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.819375 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grwcg\" (UniqueName: \"kubernetes.io/projected/3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351-kube-api-access-grwcg\") pod \"keda-admission-5fbd5c4644-jj5rv\" (UID: \"3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351\") " pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" Jul 06 20:40:52.920387 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.920362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351-certificates\") pod \"keda-admission-5fbd5c4644-jj5rv\" (UID: \"3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351\") " pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" Jul 06 20:40:52.920710 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.920425 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grwcg\" (UniqueName: \"kubernetes.io/projected/3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351-kube-api-access-grwcg\") pod \"keda-admission-5fbd5c4644-jj5rv\" (UID: \"3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351\") " pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" Jul 06 20:40:52.920710 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:52.920508 2582 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found Jul 06 20:40:52.920710 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:52.920533 2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-jj5rv: secret "keda-admission-webhooks-certs" not found Jul 06 20:40:52.920710 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:52.920584 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351-certificates podName:3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351 nodeName:}" failed. No retries permitted until 2026-07-06 20:40:53.420569636 +0000 UTC m=+320.506124825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351-certificates") pod "keda-admission-5fbd5c4644-jj5rv" (UID: "3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351") : secret "keda-admission-webhooks-certs" not found Jul 06 20:40:52.933691 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:52.933662 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grwcg\" (UniqueName: \"kubernetes.io/projected/3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351-kube-api-access-grwcg\") pod \"keda-admission-5fbd5c4644-jj5rv\" (UID: \"3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351\") " pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" Jul 06 20:40:53.000844 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:53.000820 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:40:53.021532 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:53.021465 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/31ae1ae7-371b-4b6c-b85a-49447ad4fbd2-certificates\") pod \"keda-metrics-apiserver-559f75f947-p9jkk\" (UID: \"31ae1ae7-371b-4b6c-b85a-49447ad4fbd2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:40:53.024595 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:53.024358 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/31ae1ae7-371b-4b6c-b85a-49447ad4fbd2-certificates\") pod \"keda-metrics-apiserver-559f75f947-p9jkk\" (UID: \"31ae1ae7-371b-4b6c-b85a-49447ad4fbd2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:40:53.151456 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:53.151429 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-x7lck"] Jul 06 20:40:53.153979 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:40:53.153948 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aac1acb_f216_4365_8ee3_6e893c3904d4.slice/crio-61488624570c080165cafdffcd1a46f62be1466797b9c166bac0d7a19116c60d WatchSource:0}: Error finding container 61488624570c080165cafdffcd1a46f62be1466797b9c166bac0d7a19116c60d: Status 404 returned error can't find the container with id 61488624570c080165cafdffcd1a46f62be1466797b9c166bac0d7a19116c60d Jul 06 20:40:53.238274 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:53.238246 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:40:53.378895 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:53.378867 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk"] Jul 06 20:40:53.381613 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:40:53.381583 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ae1ae7_371b_4b6c_b85a_49447ad4fbd2.slice/crio-9cbb2c5933f737d14643c5c5743525c4f9306a376d97008d676ce617d5fe941b WatchSource:0}: Error finding container 9cbb2c5933f737d14643c5c5743525c4f9306a376d97008d676ce617d5fe941b: Status 404 returned error can't find the container with id 9cbb2c5933f737d14643c5c5743525c4f9306a376d97008d676ce617d5fe941b Jul 06 20:40:53.424340 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:53.424318 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351-certificates\") pod \"keda-admission-5fbd5c4644-jj5rv\" (UID: \"3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351\") " pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" Jul 06 20:40:53.424452 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:53.424440 2582 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found Jul 06 20:40:53.424493 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:53.424461 2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-jj5rv: secret "keda-admission-webhooks-certs" not found Jul 06 20:40:53.424525 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:40:53.424502 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351-certificates podName:3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351 nodeName:}" failed. No retries permitted until 2026-07-06 20:40:54.424489292 +0000 UTC m=+321.510044463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351-certificates") pod "keda-admission-5fbd5c4644-jj5rv" (UID: "3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351") : secret "keda-admission-webhooks-certs" not found Jul 06 20:40:53.902393 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:53.902355 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-x7lck" event={"ID":"1aac1acb-f216-4365-8ee3-6e893c3904d4","Type":"ContainerStarted","Data":"61488624570c080165cafdffcd1a46f62be1466797b9c166bac0d7a19116c60d"} Jul 06 20:40:53.903403 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:53.903370 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" event={"ID":"31ae1ae7-371b-4b6c-b85a-49447ad4fbd2","Type":"ContainerStarted","Data":"9cbb2c5933f737d14643c5c5743525c4f9306a376d97008d676ce617d5fe941b"} Jul 06 20:40:54.432969 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:54.432936 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351-certificates\") pod \"keda-admission-5fbd5c4644-jj5rv\" (UID: \"3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351\") " pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" Jul 06 20:40:54.435814 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:54.435449 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351-certificates\") pod \"keda-admission-5fbd5c4644-jj5rv\" (UID: \"3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351\") " pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" Jul 06 20:40:54.562625 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:54.562598 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" Jul 06 20:40:54.733913 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:54.733879 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-jj5rv"] Jul 06 20:40:54.909830 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:54.909792 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" event={"ID":"3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351","Type":"ContainerStarted","Data":"f245960d7087de7fb73034b1daac7a0f1585543f3439d49fb1d07f2c0bc4f8c5"} Jul 06 20:40:58.929105 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:58.929067 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" event={"ID":"31ae1ae7-371b-4b6c-b85a-49447ad4fbd2","Type":"ContainerStarted","Data":"790552ee8832546b66771b2293fbe6fa8cfad5f7d876edefa3aaa9d2d96b3b50"} Jul 06 20:40:58.929495 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:58.929185 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:40:58.930545 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:58.930521 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-x7lck" event={"ID":"1aac1acb-f216-4365-8ee3-6e893c3904d4","Type":"ContainerStarted","Data":"60dd70c157c137ed8eb4fbe4b27dcd1e54d23be1a9d03fbef7e5b6eb2b958340"} Jul 06 20:40:58.930660 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:58.930646 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:40:58.949110 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:58.949066 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" podStartSLOduration=2.332115227 podStartE2EDuration="6.949054712s" totalImagesPullingTime="4.616939485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:40:58.948108506 +0000 UTC m=+326.033663698" watchObservedRunningTime="2026-07-06 20:40:58.949054712 +0000 UTC m=+326.034609904" Jul 06 20:40:58.966546 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:40:58.966506 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-x7lck" podStartSLOduration=2.118349378 podStartE2EDuration="6.966494251s" totalImagesPullingTime="4.848144873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:40:58.966040054 +0000 UTC m=+326.051595247" watchObservedRunningTime="2026-07-06 20:40:58.966494251 +0000 UTC m=+326.052049443" Jul 06 20:41:01.949102 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:41:01.949070 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" event={"ID":"3ad4aab1-6d8c-4362-a8ee-bf3df5ec4351","Type":"ContainerStarted","Data":"ada8af3c859840d8ccf5bd7dae2c41b34795ad6cbc0140bcaf308604db699371"} Jul 06 20:41:01.949475 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:41:01.949159 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" Jul 06 20:41:01.970583 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:41:01.970540 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" podStartSLOduration=3.288507512 podStartE2EDuration="9.970527519s" totalImagesPullingTime="6.682020007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:41:01.969709647 +0000 UTC m=+329.055264840" watchObservedRunningTime="2026-07-06 20:41:01.970527519 +0000 UTC m=+329.056082721" Jul 06 20:41:09.938617 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:41:09.938588 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-p9jkk" Jul 06 20:41:12.899369 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:41:12.899342 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-dhw4h" Jul 06 20:41:19.936578 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:41:19.936549 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-x7lck" Jul 06 20:41:22.955112 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:41:22.955076 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-jj5rv" Jul 06 20:42:03.127309 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.127279 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-9ctjx"] Jul 06 20:42:03.129474 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.129456 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-9ctjx" Jul 06 20:42:03.132257 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.132228 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 06 20:42:03.132378 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.132315 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 06 20:42:03.133630 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.133616 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6xp9f\"" Jul 06 20:42:03.133679 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.133650 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 06 20:42:03.143747 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.143724 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-9ctjx"] Jul 06 20:42:03.228529 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.228496 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkjpt\" (UniqueName: \"kubernetes.io/projected/3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f-kube-api-access-fkjpt\") pod \"seaweedfs-64568bcd49-9ctjx\" (UID: \"3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f\") " pod="kserve/seaweedfs-64568bcd49-9ctjx" Jul 06 20:42:03.228690 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.228548 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f-data\") pod \"seaweedfs-64568bcd49-9ctjx\" (UID: \"3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f\") " pod="kserve/seaweedfs-64568bcd49-9ctjx" Jul 06 20:42:03.329588 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.329550 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjpt\" (UniqueName: \"kubernetes.io/projected/3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f-kube-api-access-fkjpt\") pod \"seaweedfs-64568bcd49-9ctjx\" (UID: \"3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f\") " pod="kserve/seaweedfs-64568bcd49-9ctjx" Jul 06 20:42:03.329801 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.329717 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f-data\") pod \"seaweedfs-64568bcd49-9ctjx\" (UID: \"3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f\") " pod="kserve/seaweedfs-64568bcd49-9ctjx" Jul 06 20:42:03.330107 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.330090 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f-data\") pod \"seaweedfs-64568bcd49-9ctjx\" (UID: \"3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f\") " pod="kserve/seaweedfs-64568bcd49-9ctjx" Jul 06 20:42:03.342359 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.342344 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjpt\" (UniqueName: \"kubernetes.io/projected/3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f-kube-api-access-fkjpt\") pod \"seaweedfs-64568bcd49-9ctjx\" (UID: \"3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f\") " pod="kserve/seaweedfs-64568bcd49-9ctjx" Jul 06 20:42:03.438329 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.438250 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-9ctjx" Jul 06 20:42:03.570782 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:03.570734 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-9ctjx"] Jul 06 20:42:03.574027 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:42:03.574001 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad1a5b4_dbce_4b1f_b2cb_e40c48bbd61f.slice/crio-18a327f9e97b01a0efa5133ed2de90cf088578e163f0873b05bfcd04466a5470 WatchSource:0}: Error finding container 18a327f9e97b01a0efa5133ed2de90cf088578e163f0873b05bfcd04466a5470: Status 404 returned error can't find the container with id 18a327f9e97b01a0efa5133ed2de90cf088578e163f0873b05bfcd04466a5470 Jul 06 20:42:04.205996 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:04.205945 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-9ctjx" event={"ID":"3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f","Type":"ContainerStarted","Data":"18a327f9e97b01a0efa5133ed2de90cf088578e163f0873b05bfcd04466a5470"} Jul 06 20:42:07.220590 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:07.220556 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-9ctjx" event={"ID":"3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f","Type":"ContainerStarted","Data":"a73c5aeb39d7fe18ae84ae24f11024d2659a899b9fe333658beb2116a0c3965a"} Jul 06 20:42:07.221013 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:07.220672 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-9ctjx" Jul 06 20:42:07.244942 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:07.244878 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-9ctjx" podStartSLOduration=1.218313563 podStartE2EDuration="4.244860237s" totalImagesPullingTime="3.026546674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:42:07.243995855 +0000 UTC m=+394.329551059" watchObservedRunningTime="2026-07-06 20:42:07.244860237 +0000 UTC m=+394.330415730" Jul 06 20:42:13.228319 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:42:13.228284 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-9ctjx" Jul 06 20:43:40.499244 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.499211 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bbf69ddd9-nmzsq"] Jul 06 20:43:40.502639 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.502616 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.513798 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.513775 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbf69ddd9-nmzsq"] Jul 06 20:43:40.580323 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.580304 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xzst\" (UniqueName: \"kubernetes.io/projected/758e406a-bd93-4cc2-a19e-3b50000c7889-kube-api-access-4xzst\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.580416 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.580331 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/758e406a-bd93-4cc2-a19e-3b50000c7889-service-ca\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.580416 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.580348 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/758e406a-bd93-4cc2-a19e-3b50000c7889-console-serving-cert\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.580416 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.580366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/758e406a-bd93-4cc2-a19e-3b50000c7889-console-oauth-config\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.580416 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.580386 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/758e406a-bd93-4cc2-a19e-3b50000c7889-oauth-serving-cert\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.580605 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.580456 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/758e406a-bd93-4cc2-a19e-3b50000c7889-trusted-ca-bundle\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.580605 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.580512 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/758e406a-bd93-4cc2-a19e-3b50000c7889-console-config\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.680991 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.680963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/758e406a-bd93-4cc2-a19e-3b50000c7889-oauth-serving-cert\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.681110 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.681006 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/758e406a-bd93-4cc2-a19e-3b50000c7889-trusted-ca-bundle\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.681110 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.681040 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/758e406a-bd93-4cc2-a19e-3b50000c7889-console-config\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.681289 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.681264 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xzst\" (UniqueName: \"kubernetes.io/projected/758e406a-bd93-4cc2-a19e-3b50000c7889-kube-api-access-4xzst\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.681374 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.681314 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/758e406a-bd93-4cc2-a19e-3b50000c7889-service-ca\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.681374 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.681361 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/758e406a-bd93-4cc2-a19e-3b50000c7889-console-serving-cert\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.681471 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.681404 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/758e406a-bd93-4cc2-a19e-3b50000c7889-console-oauth-config\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.681733 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.681711 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/758e406a-bd93-4cc2-a19e-3b50000c7889-console-config\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.681848 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.681784 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/758e406a-bd93-4cc2-a19e-3b50000c7889-oauth-serving-cert\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.681901 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.681871 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/758e406a-bd93-4cc2-a19e-3b50000c7889-trusted-ca-bundle\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.681953 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.681902 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/758e406a-bd93-4cc2-a19e-3b50000c7889-service-ca\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.683734 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.683713 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/758e406a-bd93-4cc2-a19e-3b50000c7889-console-serving-cert\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.683829 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.683735 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/758e406a-bd93-4cc2-a19e-3b50000c7889-console-oauth-config\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.691876 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.691854 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xzst\" (UniqueName: \"kubernetes.io/projected/758e406a-bd93-4cc2-a19e-3b50000c7889-kube-api-access-4xzst\") pod \"console-bbf69ddd9-nmzsq\" (UID: \"758e406a-bd93-4cc2-a19e-3b50000c7889\") " pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.811659 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.811614 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:40.936948 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:40.936929 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbf69ddd9-nmzsq"] Jul 06 20:43:40.939188 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:43:40.939162 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758e406a_bd93_4cc2_a19e_3b50000c7889.slice/crio-982e9d7e18514698861ca88df231a51c007f1963f187df485ee33efe6fb275d9 WatchSource:0}: Error finding container 982e9d7e18514698861ca88df231a51c007f1963f187df485ee33efe6fb275d9: Status 404 returned error can't find the container with id 982e9d7e18514698861ca88df231a51c007f1963f187df485ee33efe6fb275d9 Jul 06 20:43:41.605610 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:41.605576 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbf69ddd9-nmzsq" event={"ID":"758e406a-bd93-4cc2-a19e-3b50000c7889","Type":"ContainerStarted","Data":"2c33f873a6b6783bddbe24be97362d86940dfc0a562a572b9f256f8d69f0220e"} Jul 06 20:43:41.605993 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:41.605618 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbf69ddd9-nmzsq" event={"ID":"758e406a-bd93-4cc2-a19e-3b50000c7889","Type":"ContainerStarted","Data":"982e9d7e18514698861ca88df231a51c007f1963f187df485ee33efe6fb275d9"} Jul 06 20:43:41.626755 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:41.626707 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bbf69ddd9-nmzsq" podStartSLOduration=1.626693572 podStartE2EDuration="1.626693572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:43:41.626203888 +0000 UTC m=+488.711759104" watchObservedRunningTime="2026-07-06 20:43:41.626693572 +0000 UTC m=+488.712248765" Jul 06 20:43:50.812067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:50.812037 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:50.812598 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:50.812090 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:50.816505 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:50.816485 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:51.647633 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:51.647608 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bbf69ddd9-nmzsq" Jul 06 20:43:51.708781 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:43:51.708722 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d8449695-59jzs"] Jul 06 20:44:16.736216 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:16.736172 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-54d8449695-59jzs" podUID="35231930-75b0-4f52-bf5c-fee7cc8990c9" containerName="console" containerID="cri-o://ce727af982b11993554e241878a78fdaa2df84255f09a8f87f778bd568a287c6" gracePeriod=15 Jul 06 20:44:16.971671 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:16.971649 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:44:17.045275 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.045213 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-config\") pod \"35231930-75b0-4f52-bf5c-fee7cc8990c9\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " Jul 06 20:44:17.045275 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.045251 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-trusted-ca-bundle\") pod \"35231930-75b0-4f52-bf5c-fee7cc8990c9\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " Jul 06 20:44:17.045448 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.045280 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-serving-cert\") pod \"35231930-75b0-4f52-bf5c-fee7cc8990c9\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " Jul 06 20:44:17.045448 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.045311 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-oauth-config\") pod \"35231930-75b0-4f52-bf5c-fee7cc8990c9\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " Jul 06 20:44:17.045448 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.045344 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-service-ca\") pod \"35231930-75b0-4f52-bf5c-fee7cc8990c9\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " Jul 06 20:44:17.045448 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.045376 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqmqh\" (UniqueName: \"kubernetes.io/projected/35231930-75b0-4f52-bf5c-fee7cc8990c9-kube-api-access-sqmqh\") pod \"35231930-75b0-4f52-bf5c-fee7cc8990c9\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " Jul 06 20:44:17.045448 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.045421 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-oauth-serving-cert\") pod \"35231930-75b0-4f52-bf5c-fee7cc8990c9\" (UID: \"35231930-75b0-4f52-bf5c-fee7cc8990c9\") " Jul 06 20:44:17.045686 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.045649 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "35231930-75b0-4f52-bf5c-fee7cc8990c9" (UID: "35231930-75b0-4f52-bf5c-fee7cc8990c9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:44:17.045878 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.045852 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-config" (OuterVolumeSpecName: "console-config") pod "35231930-75b0-4f52-bf5c-fee7cc8990c9" (UID: "35231930-75b0-4f52-bf5c-fee7cc8990c9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:44:17.045975 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.045888 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "35231930-75b0-4f52-bf5c-fee7cc8990c9" (UID: "35231930-75b0-4f52-bf5c-fee7cc8990c9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:44:17.045975 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.045907 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-service-ca" (OuterVolumeSpecName: "service-ca") pod "35231930-75b0-4f52-bf5c-fee7cc8990c9" (UID: "35231930-75b0-4f52-bf5c-fee7cc8990c9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:44:17.047584 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.047553 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "35231930-75b0-4f52-bf5c-fee7cc8990c9" (UID: "35231930-75b0-4f52-bf5c-fee7cc8990c9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:44:17.047667 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.047578 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "35231930-75b0-4f52-bf5c-fee7cc8990c9" (UID: "35231930-75b0-4f52-bf5c-fee7cc8990c9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:44:17.047667 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.047652 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35231930-75b0-4f52-bf5c-fee7cc8990c9-kube-api-access-sqmqh" (OuterVolumeSpecName: "kube-api-access-sqmqh") pod "35231930-75b0-4f52-bf5c-fee7cc8990c9" (UID: "35231930-75b0-4f52-bf5c-fee7cc8990c9"). InnerVolumeSpecName "kube-api-access-sqmqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:44:17.146328 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.146289 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqmqh\" (UniqueName: \"kubernetes.io/projected/35231930-75b0-4f52-bf5c-fee7cc8990c9-kube-api-access-sqmqh\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:44:17.146328 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.146323 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-oauth-serving-cert\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:44:17.146476 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.146340 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:44:17.146476 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.146353 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-trusted-ca-bundle\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:44:17.146476 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.146366 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-serving-cert\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:44:17.146476 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.146378 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35231930-75b0-4f52-bf5c-fee7cc8990c9-console-oauth-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:44:17.146476 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.146389 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35231930-75b0-4f52-bf5c-fee7cc8990c9-service-ca\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:44:17.576836 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.576808 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-zcnd4"] Jul 06 20:44:17.577123 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.577110 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35231930-75b0-4f52-bf5c-fee7cc8990c9" containerName="console" Jul 06 20:44:17.577165 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.577124 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="35231930-75b0-4f52-bf5c-fee7cc8990c9" containerName="console" Jul 06 20:44:17.577198 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.577190 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="35231930-75b0-4f52-bf5c-fee7cc8990c9" containerName="console" Jul 06 20:44:17.580356 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.580339 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-zcnd4" Jul 06 20:44:17.586580 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.586422 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Jul 06 20:44:17.586580 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.586436 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-4fdr4\"" Jul 06 20:44:17.621195 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.621173 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-zcnd4"] Jul 06 20:44:17.649800 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.649751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb92m\" (UniqueName: \"kubernetes.io/projected/8dbea9d6-ed55-45cd-93bd-e5541f8374b5-kube-api-access-zb92m\") pod \"model-serving-api-fd65d7d6-zcnd4\" (UID: \"8dbea9d6-ed55-45cd-93bd-e5541f8374b5\") " pod="kserve/model-serving-api-fd65d7d6-zcnd4" Jul 06 20:44:17.649905 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.649863 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbea9d6-ed55-45cd-93bd-e5541f8374b5-tls-certs\") pod \"model-serving-api-fd65d7d6-zcnd4\" (UID: \"8dbea9d6-ed55-45cd-93bd-e5541f8374b5\") " pod="kserve/model-serving-api-fd65d7d6-zcnd4" Jul 06 20:44:17.750872 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.750848 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbea9d6-ed55-45cd-93bd-e5541f8374b5-tls-certs\") pod \"model-serving-api-fd65d7d6-zcnd4\" (UID: \"8dbea9d6-ed55-45cd-93bd-e5541f8374b5\") " pod="kserve/model-serving-api-fd65d7d6-zcnd4" Jul 06 20:44:17.751194 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.750894 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb92m\" (UniqueName: \"kubernetes.io/projected/8dbea9d6-ed55-45cd-93bd-e5541f8374b5-kube-api-access-zb92m\") pod \"model-serving-api-fd65d7d6-zcnd4\" (UID: \"8dbea9d6-ed55-45cd-93bd-e5541f8374b5\") " pod="kserve/model-serving-api-fd65d7d6-zcnd4" Jul 06 20:44:17.751194 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:44:17.750973 2582 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 06 20:44:17.751194 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:44:17.751051 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbea9d6-ed55-45cd-93bd-e5541f8374b5-tls-certs podName:8dbea9d6-ed55-45cd-93bd-e5541f8374b5 nodeName:}" failed. No retries permitted until 2026-07-06 20:44:18.251035086 +0000 UTC m=+525.336590261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8dbea9d6-ed55-45cd-93bd-e5541f8374b5-tls-certs") pod "model-serving-api-fd65d7d6-zcnd4" (UID: "8dbea9d6-ed55-45cd-93bd-e5541f8374b5") : secret "model-serving-api-tls" not found Jul 06 20:44:17.755335 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.755315 2582 generic.go:356] "Generic (PLEG): container finished" podID="35231930-75b0-4f52-bf5c-fee7cc8990c9" containerID="ce727af982b11993554e241878a78fdaa2df84255f09a8f87f778bd568a287c6" exitCode=0 Jul 06 20:44:17.755427 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.755347 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d8449695-59jzs" event={"ID":"35231930-75b0-4f52-bf5c-fee7cc8990c9","Type":"ContainerDied","Data":"ce727af982b11993554e241878a78fdaa2df84255f09a8f87f778bd568a287c6"} Jul 06 20:44:17.755427 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.755362 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d8449695-59jzs" Jul 06 20:44:17.755427 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.755392 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d8449695-59jzs" event={"ID":"35231930-75b0-4f52-bf5c-fee7cc8990c9","Type":"ContainerDied","Data":"a5a3e37dffdaf7df3ce4370a7b04f57ecba7d1aedb997c52184d369d3b745c4b"} Jul 06 20:44:17.755427 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.755413 2582 scope.go:117] "RemoveContainer" containerID="ce727af982b11993554e241878a78fdaa2df84255f09a8f87f778bd568a287c6" Jul 06 20:44:17.767494 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.767466 2582 scope.go:117] "RemoveContainer" containerID="ce727af982b11993554e241878a78fdaa2df84255f09a8f87f778bd568a287c6" Jul 06 20:44:17.767847 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:44:17.767819 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce727af982b11993554e241878a78fdaa2df84255f09a8f87f778bd568a287c6\": container with ID starting with ce727af982b11993554e241878a78fdaa2df84255f09a8f87f778bd568a287c6 not found: ID does not exist" containerID="ce727af982b11993554e241878a78fdaa2df84255f09a8f87f778bd568a287c6" Jul 06 20:44:17.767920 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.767855 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce727af982b11993554e241878a78fdaa2df84255f09a8f87f778bd568a287c6"} err="failed to get container status \"ce727af982b11993554e241878a78fdaa2df84255f09a8f87f778bd568a287c6\": rpc error: code = NotFound desc = could not find container \"ce727af982b11993554e241878a78fdaa2df84255f09a8f87f778bd568a287c6\": container with ID starting with ce727af982b11993554e241878a78fdaa2df84255f09a8f87f778bd568a287c6 not found: ID does not exist" Jul 06 20:44:17.771896 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.771877 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb92m\" (UniqueName: \"kubernetes.io/projected/8dbea9d6-ed55-45cd-93bd-e5541f8374b5-kube-api-access-zb92m\") pod \"model-serving-api-fd65d7d6-zcnd4\" (UID: \"8dbea9d6-ed55-45cd-93bd-e5541f8374b5\") " pod="kserve/model-serving-api-fd65d7d6-zcnd4" Jul 06 20:44:17.788393 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.788373 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d8449695-59jzs"] Jul 06 20:44:17.798027 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:17.797982 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54d8449695-59jzs"] Jul 06 20:44:18.254105 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:18.254077 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbea9d6-ed55-45cd-93bd-e5541f8374b5-tls-certs\") pod \"model-serving-api-fd65d7d6-zcnd4\" (UID: \"8dbea9d6-ed55-45cd-93bd-e5541f8374b5\") " pod="kserve/model-serving-api-fd65d7d6-zcnd4" Jul 06 20:44:18.256304 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:18.256282 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dbea9d6-ed55-45cd-93bd-e5541f8374b5-tls-certs\") pod \"model-serving-api-fd65d7d6-zcnd4\" (UID: \"8dbea9d6-ed55-45cd-93bd-e5541f8374b5\") " pod="kserve/model-serving-api-fd65d7d6-zcnd4" Jul 06 20:44:18.490148 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:18.490113 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-zcnd4" Jul 06 20:44:18.621860 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:18.621832 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-zcnd4"] Jul 06 20:44:18.624998 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:44:18.624973 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbea9d6_ed55_45cd_93bd_e5541f8374b5.slice/crio-7999af1f233e8ab2049b556f180b422ea780a468322e4b47ad66301b17d732fb WatchSource:0}: Error finding container 7999af1f233e8ab2049b556f180b422ea780a468322e4b47ad66301b17d732fb: Status 404 returned error can't find the container with id 7999af1f233e8ab2049b556f180b422ea780a468322e4b47ad66301b17d732fb Jul 06 20:44:18.761311 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:18.761278 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-zcnd4" event={"ID":"8dbea9d6-ed55-45cd-93bd-e5541f8374b5","Type":"ContainerStarted","Data":"7999af1f233e8ab2049b556f180b422ea780a468322e4b47ad66301b17d732fb"} Jul 06 20:44:19.540905 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:19.540850 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35231930-75b0-4f52-bf5c-fee7cc8990c9" path="/var/lib/kubelet/pods/35231930-75b0-4f52-bf5c-fee7cc8990c9/volumes" Jul 06 20:44:21.779128 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:21.779095 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-zcnd4" event={"ID":"8dbea9d6-ed55-45cd-93bd-e5541f8374b5","Type":"ContainerStarted","Data":"b0a9e1d14164c95450dca4c8d6caf2791647fd729f171fccdaebe100430912b8"} Jul 06 20:44:21.779649 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:21.779188 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-zcnd4" Jul 06 20:44:21.800372 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:21.800324 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-zcnd4" podStartSLOduration=2.552889553 podStartE2EDuration="4.800309339s" totalImagesPullingTime="2.247419786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:44:21.799336888 +0000 UTC m=+528.884892073" watchObservedRunningTime="2026-07-06 20:44:21.800309339 +0000 UTC m=+528.885864534" Jul 06 20:44:32.786947 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:44:32.786873 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-zcnd4" Jul 06 20:45:33.362958 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:45:33.362924 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 20:45:33.364365 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:45:33.364341 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 20:46:04.614898 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.614822 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k"] Jul 06 20:46:04.618155 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.618133 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:04.626678 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.626655 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-85420-predictor-serving-cert\"" Jul 06 20:46:04.627373 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.627353 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-85420-kube-rbac-proxy-sar-config\"" Jul 06 20:46:04.628251 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.628236 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-b6hg7\"" Jul 06 20:46:04.628319 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.628274 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 06 20:46:04.628375 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.628357 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 06 20:46:04.635388 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.635364 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k"] Jul 06 20:46:04.669129 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.669097 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-85420-predictor-8c694c98b-cnn9k\" (UID: \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\") " pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:04.669247 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.669145 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-proxy-tls\") pod \"success-200-isvc-85420-predictor-8c694c98b-cnn9k\" (UID: \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\") " pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:04.669247 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.669173 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqstn\" (UniqueName: \"kubernetes.io/projected/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-kube-api-access-cqstn\") pod \"success-200-isvc-85420-predictor-8c694c98b-cnn9k\" (UID: \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\") " pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:04.769507 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.769477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-proxy-tls\") pod \"success-200-isvc-85420-predictor-8c694c98b-cnn9k\" (UID: \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\") " pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:04.769667 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.769517 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqstn\" (UniqueName: \"kubernetes.io/projected/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-kube-api-access-cqstn\") pod \"success-200-isvc-85420-predictor-8c694c98b-cnn9k\" (UID: \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\") " pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:04.769667 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.769559 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-85420-predictor-8c694c98b-cnn9k\" (UID: \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\") " pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:04.769667 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:46:04.769629 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-85420-predictor-serving-cert: secret "success-200-isvc-85420-predictor-serving-cert" not found Jul 06 20:46:04.769865 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:46:04.769684 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-proxy-tls podName:601c9d0f-a9df-4b87-942a-173b3ac9d8b2 nodeName:}" failed. No retries permitted until 2026-07-06 20:46:05.269667144 +0000 UTC m=+632.355222321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-proxy-tls") pod "success-200-isvc-85420-predictor-8c694c98b-cnn9k" (UID: "601c9d0f-a9df-4b87-942a-173b3ac9d8b2") : secret "success-200-isvc-85420-predictor-serving-cert" not found Jul 06 20:46:04.770223 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.770204 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-85420-predictor-8c694c98b-cnn9k\" (UID: \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\") " pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:04.778970 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:04.778946 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqstn\" (UniqueName: \"kubernetes.io/projected/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-kube-api-access-cqstn\") pod \"success-200-isvc-85420-predictor-8c694c98b-cnn9k\" (UID: \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\") " pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:05.006133 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.006042 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd"] Jul 06 20:46:05.009636 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.009614 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:05.012705 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.012682 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-85420-predictor-serving-cert\"" Jul 06 20:46:05.012705 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.012696 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-85420-kube-rbac-proxy-sar-config\"" Jul 06 20:46:05.019893 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.019869 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd"] Jul 06 20:46:05.071399 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.071365 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-proxy-tls\") pod \"error-404-isvc-85420-predictor-856b7d987-9rsjd\" (UID: \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\") " pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:05.071603 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.071420 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4jst\" (UniqueName: \"kubernetes.io/projected/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-kube-api-access-v4jst\") pod \"error-404-isvc-85420-predictor-856b7d987-9rsjd\" (UID: \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\") " pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:05.071603 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.071495 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-85420-predictor-856b7d987-9rsjd\" (UID: \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\") " pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:05.172585 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.172551 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-85420-predictor-856b7d987-9rsjd\" (UID: \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\") " pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:05.172744 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.172607 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-proxy-tls\") pod \"error-404-isvc-85420-predictor-856b7d987-9rsjd\" (UID: \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\") " pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:05.172744 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.172724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4jst\" (UniqueName: \"kubernetes.io/projected/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-kube-api-access-v4jst\") pod \"error-404-isvc-85420-predictor-856b7d987-9rsjd\" (UID: \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\") " pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:05.172894 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:46:05.172860 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-85420-predictor-serving-cert: secret "error-404-isvc-85420-predictor-serving-cert" not found Jul 06 20:46:05.172956 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:46:05.172942 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-proxy-tls podName:ae9160a2-4983-46e5-aa8c-c2ae98c0af05 nodeName:}" failed. No retries permitted until 2026-07-06 20:46:05.672908737 +0000 UTC m=+632.758463932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-proxy-tls") pod "error-404-isvc-85420-predictor-856b7d987-9rsjd" (UID: "ae9160a2-4983-46e5-aa8c-c2ae98c0af05") : secret "error-404-isvc-85420-predictor-serving-cert" not found Jul 06 20:46:05.173478 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.173455 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-85420-predictor-856b7d987-9rsjd\" (UID: \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\") " pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:05.186364 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.186338 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4jst\" (UniqueName: \"kubernetes.io/projected/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-kube-api-access-v4jst\") pod \"error-404-isvc-85420-predictor-856b7d987-9rsjd\" (UID: \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\") " pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:05.273755 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.273676 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-proxy-tls\") pod \"success-200-isvc-85420-predictor-8c694c98b-cnn9k\" (UID: \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\") " pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:05.276178 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.276154 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-proxy-tls\") pod \"success-200-isvc-85420-predictor-8c694c98b-cnn9k\" (UID: \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\") " pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:05.528510 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.528421 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:05.666594 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.666557 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k"] Jul 06 20:46:05.670995 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:46:05.670969 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601c9d0f_a9df_4b87_942a_173b3ac9d8b2.slice/crio-a49fe4214ea0fcc0894d8e8a25ec30005bf61dce9c63cfe41d48e57b468adc8d WatchSource:0}: Error finding container a49fe4214ea0fcc0894d8e8a25ec30005bf61dce9c63cfe41d48e57b468adc8d: Status 404 returned error can't find the container with id a49fe4214ea0fcc0894d8e8a25ec30005bf61dce9c63cfe41d48e57b468adc8d Jul 06 20:46:05.672914 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.672899 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:46:05.677713 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.677694 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-proxy-tls\") pod \"error-404-isvc-85420-predictor-856b7d987-9rsjd\" (UID: \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\") " pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:05.680036 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.680009 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-proxy-tls\") pod \"error-404-isvc-85420-predictor-856b7d987-9rsjd\" (UID: \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\") " pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:05.920942 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:05.920891 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:06.057648 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:06.057620 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd"] Jul 06 20:46:06.059592 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:46:06.059563 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9160a2_4983_46e5_aa8c_c2ae98c0af05.slice/crio-48dde3772c5cc070ea440a6d720d20e4f06780e1c54b63b4601a188d6cba0e3d WatchSource:0}: Error finding container 48dde3772c5cc070ea440a6d720d20e4f06780e1c54b63b4601a188d6cba0e3d: Status 404 returned error can't find the container with id 48dde3772c5cc070ea440a6d720d20e4f06780e1c54b63b4601a188d6cba0e3d Jul 06 20:46:06.233116 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:06.233022 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" event={"ID":"ae9160a2-4983-46e5-aa8c-c2ae98c0af05","Type":"ContainerStarted","Data":"48dde3772c5cc070ea440a6d720d20e4f06780e1c54b63b4601a188d6cba0e3d"} Jul 06 20:46:06.234515 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:06.234483 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" event={"ID":"601c9d0f-a9df-4b87-942a-173b3ac9d8b2","Type":"ContainerStarted","Data":"a49fe4214ea0fcc0894d8e8a25ec30005bf61dce9c63cfe41d48e57b468adc8d"} Jul 06 20:46:22.341737 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:22.341639 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" event={"ID":"ae9160a2-4983-46e5-aa8c-c2ae98c0af05","Type":"ContainerStarted","Data":"c9770be4d2de2f9f58ac7ea52ba55dac7483c4fd8d3a9d32fd257d4c01179d74"} Jul 06 20:46:22.345723 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:22.345644 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" event={"ID":"601c9d0f-a9df-4b87-942a-173b3ac9d8b2","Type":"ContainerStarted","Data":"c80d12666aa62754540ba588ed2ef9d1829d511e4cee75bc5ee75ca12faa823c"} Jul 06 20:46:24.356792 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:24.356726 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" event={"ID":"ae9160a2-4983-46e5-aa8c-c2ae98c0af05","Type":"ContainerStarted","Data":"b732d08c2555736478fa0930ea851b19dc6aec88f4e363277ceced57a44911af"} Jul 06 20:46:24.357261 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:24.356968 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:24.357261 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:24.357025 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:24.358337 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:24.358302 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 06 20:46:24.358595 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:24.358571 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" event={"ID":"601c9d0f-a9df-4b87-942a-173b3ac9d8b2","Type":"ContainerStarted","Data":"756d03b3971a2a0545736b17e37fb16259853349bac4394908c4044fb24b4413"} Jul 06 20:46:24.358716 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:24.358702 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:24.358779 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:24.358748 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:24.359657 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:24.359639 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 06 20:46:24.386531 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:24.386483 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" podStartSLOduration=2.548196275 podStartE2EDuration="20.386467489s" totalImagesPullingTime="17.838271214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:46:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:46:24.38341654 +0000 UTC m=+651.468971735" watchObservedRunningTime="2026-07-06 20:46:24.386467489 +0000 UTC m=+651.472022683" Jul 06 20:46:24.406471 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:24.406428 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" podStartSLOduration=2.195962332 podStartE2EDuration="20.406415248s" totalImagesPullingTime="18.210452916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:46:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:46:24.404780869 +0000 UTC m=+651.490336058" watchObservedRunningTime="2026-07-06 20:46:24.406415248 +0000 UTC m=+651.491970442" Jul 06 20:46:25.362619 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:25.362570 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 06 20:46:25.363102 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:25.362636 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 06 20:46:30.367233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:30.367204 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:46:30.367739 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:30.367496 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:46:30.367739 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:30.367682 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 06 20:46:30.368137 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:30.368115 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 06 20:46:40.368177 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:40.368139 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 06 20:46:40.368657 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:40.368144 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 06 20:46:50.368004 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:50.367962 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 06 20:46:50.368484 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:46:50.368115 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 06 20:47:00.368280 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:00.368242 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Jul 06 20:47:00.368653 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:00.368241 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 06 20:47:10.368947 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:10.368906 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:47:10.369398 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:10.369001 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:47:34.979539 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:34.979452 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k"] Jul 06 20:47:34.980094 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:34.979688 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kserve-container" containerID="cri-o://c80d12666aa62754540ba588ed2ef9d1829d511e4cee75bc5ee75ca12faa823c" gracePeriod=30 Jul 06 20:47:34.980094 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:34.979732 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kube-rbac-proxy" containerID="cri-o://756d03b3971a2a0545736b17e37fb16259853349bac4394908c4044fb24b4413" gracePeriod=30 Jul 06 20:47:35.033391 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.033360 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2"] Jul 06 20:47:35.035810 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.035794 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:35.041880 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.041823 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-87d4d-kube-rbac-proxy-sar-config\"" Jul 06 20:47:35.042101 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.042076 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-87d4d-predictor-serving-cert\"" Jul 06 20:47:35.053673 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.053649 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2"] Jul 06 20:47:35.133953 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.133922 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd"] Jul 06 20:47:35.134173 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.134131 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kserve-container" containerID="cri-o://c9770be4d2de2f9f58ac7ea52ba55dac7483c4fd8d3a9d32fd257d4c01179d74" gracePeriod=30 Jul 06 20:47:35.134235 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.134172 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kube-rbac-proxy" containerID="cri-o://b732d08c2555736478fa0930ea851b19dc6aec88f4e363277ceced57a44911af" gracePeriod=30 Jul 06 20:47:35.148953 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.148924 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2\" (UID: \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\") " pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:35.149045 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.148983 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-proxy-tls\") pod \"success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2\" (UID: \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\") " pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:35.149045 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.149035 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptlhf\" (UniqueName: \"kubernetes.io/projected/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-kube-api-access-ptlhf\") pod \"success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2\" (UID: \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\") " pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:35.162631 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.162606 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c"] Jul 06 20:47:35.165357 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.165335 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:35.167977 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.167955 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-87d4d-kube-rbac-proxy-sar-config\"" Jul 06 20:47:35.168085 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.167955 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-87d4d-predictor-serving-cert\"" Jul 06 20:47:35.178459 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.178435 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c"] Jul 06 20:47:35.249456 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.249428 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-proxy-tls\") pod \"success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2\" (UID: \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\") " pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:35.249563 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.249478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx5gf\" (UniqueName: \"kubernetes.io/projected/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-kube-api-access-jx5gf\") pod \"error-404-isvc-87d4d-predictor-779b4f5579-8cp6c\" (UID: \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\") " pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:35.249563 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.249501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-87d4d-predictor-779b4f5579-8cp6c\" (UID: \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\") " pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:35.249563 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.249530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptlhf\" (UniqueName: \"kubernetes.io/projected/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-kube-api-access-ptlhf\") pod \"success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2\" (UID: \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\") " pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:35.249724 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.249607 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-proxy-tls\") pod \"error-404-isvc-87d4d-predictor-779b4f5579-8cp6c\" (UID: \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\") " pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:35.249724 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.249656 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2\" (UID: \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\") " pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:35.250263 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.250244 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2\" (UID: \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\") " pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:35.251805 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.251788 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-proxy-tls\") pod \"success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2\" (UID: \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\") " pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:35.259288 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.259263 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptlhf\" (UniqueName: \"kubernetes.io/projected/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-kube-api-access-ptlhf\") pod \"success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2\" (UID: \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\") " pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:35.345383 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.345344 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:35.350644 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.350618 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx5gf\" (UniqueName: \"kubernetes.io/projected/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-kube-api-access-jx5gf\") pod \"error-404-isvc-87d4d-predictor-779b4f5579-8cp6c\" (UID: \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\") " pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:35.350726 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.350669 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-87d4d-predictor-779b4f5579-8cp6c\" (UID: \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\") " pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:35.350726 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.350714 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-proxy-tls\") pod \"error-404-isvc-87d4d-predictor-779b4f5579-8cp6c\" (UID: \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\") " pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:35.351402 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.351381 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-87d4d-predictor-779b4f5579-8cp6c\" (UID: \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\") " pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:35.353546 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.353516 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-proxy-tls\") pod \"error-404-isvc-87d4d-predictor-779b4f5579-8cp6c\" (UID: \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\") " pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:35.360651 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.360626 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx5gf\" (UniqueName: \"kubernetes.io/projected/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-kube-api-access-jx5gf\") pod \"error-404-isvc-87d4d-predictor-779b4f5579-8cp6c\" (UID: \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\") " pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:35.363684 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.363653 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused" Jul 06 20:47:35.363795 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.363680 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused" Jul 06 20:47:35.475657 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.475621 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:35.492095 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.492069 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2"] Jul 06 20:47:35.497111 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:47:35.497038 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6dd3d9_a337_45c3_a51a_218105f7ff8e.slice/crio-3c8ef28d12f8664ec4bf42b858ebf828dd072c1dac7fa0a5fba2743a72277b0f WatchSource:0}: Error finding container 3c8ef28d12f8664ec4bf42b858ebf828dd072c1dac7fa0a5fba2743a72277b0f: Status 404 returned error can't find the container with id 3c8ef28d12f8664ec4bf42b858ebf828dd072c1dac7fa0a5fba2743a72277b0f Jul 06 20:47:35.623405 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.623382 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c"] Jul 06 20:47:35.625499 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:47:35.625472 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e7d7b2_3e68_46e5_ab5f_80aed349e5f3.slice/crio-32e18378ee1fe7d4a1043ddc1d7232d743d78a1786692441ef556dbc313f36c0 WatchSource:0}: Error finding container 32e18378ee1fe7d4a1043ddc1d7232d743d78a1786692441ef556dbc313f36c0: Status 404 returned error can't find the container with id 32e18378ee1fe7d4a1043ddc1d7232d743d78a1786692441ef556dbc313f36c0 Jul 06 20:47:35.688690 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.688643 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" event={"ID":"ff6dd3d9-a337-45c3-a51a-218105f7ff8e","Type":"ContainerStarted","Data":"9d983b28894a59129f6c4baecec5c514f8e729e0d666dab23f74363469909f8c"} Jul 06 20:47:35.688823 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.688710 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" event={"ID":"ff6dd3d9-a337-45c3-a51a-218105f7ff8e","Type":"ContainerStarted","Data":"fdbdac8e2a5dfd934a389d00c54256f2ee31ae0e63941405e3a8ac75a479d3cf"} Jul 06 20:47:35.688823 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.688742 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" event={"ID":"ff6dd3d9-a337-45c3-a51a-218105f7ff8e","Type":"ContainerStarted","Data":"3c8ef28d12f8664ec4bf42b858ebf828dd072c1dac7fa0a5fba2743a72277b0f"} Jul 06 20:47:35.688823 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.688791 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:35.688823 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.688824 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:35.690488 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.690444 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Jul 06 20:47:35.690786 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.690746 2582 generic.go:356] "Generic (PLEG): container finished" podID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerID="b732d08c2555736478fa0930ea851b19dc6aec88f4e363277ceced57a44911af" exitCode=2 Jul 06 20:47:35.690864 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.690797 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" event={"ID":"ae9160a2-4983-46e5-aa8c-c2ae98c0af05","Type":"ContainerDied","Data":"b732d08c2555736478fa0930ea851b19dc6aec88f4e363277ceced57a44911af"} Jul 06 20:47:35.692651 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.692627 2582 generic.go:356] "Generic (PLEG): container finished" podID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerID="756d03b3971a2a0545736b17e37fb16259853349bac4394908c4044fb24b4413" exitCode=2 Jul 06 20:47:35.692752 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.692687 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" event={"ID":"601c9d0f-a9df-4b87-942a-173b3ac9d8b2","Type":"ContainerDied","Data":"756d03b3971a2a0545736b17e37fb16259853349bac4394908c4044fb24b4413"} Jul 06 20:47:35.694368 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.694347 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" event={"ID":"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3","Type":"ContainerStarted","Data":"88c9b2abcb11c8ce1ffdc192b2c23f4a344a4def752699543c0e2d73cb5b5b59"} Jul 06 20:47:35.694455 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.694392 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" event={"ID":"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3","Type":"ContainerStarted","Data":"32e18378ee1fe7d4a1043ddc1d7232d743d78a1786692441ef556dbc313f36c0"} Jul 06 20:47:35.720752 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:35.720713 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" podStartSLOduration=1.72070063 podStartE2EDuration="1.72070063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:47:35.718650103 +0000 UTC m=+722.804205299" watchObservedRunningTime="2026-07-06 20:47:35.72070063 +0000 UTC m=+722.806255824" Jul 06 20:47:36.700398 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:36.700361 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" event={"ID":"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3","Type":"ContainerStarted","Data":"df8a12397de10060af8fe855f43153cb1ef6d9562d60c94cf9ea6e136bfdb6e5"} Jul 06 20:47:36.700826 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:36.700673 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:36.700826 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:36.700672 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Jul 06 20:47:36.700826 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:36.700722 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:36.702166 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:36.702142 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 06 20:47:37.705524 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:37.705478 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 06 20:47:38.227427 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.227397 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:47:38.256464 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.256418 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" podStartSLOduration=3.256401462 podStartE2EDuration="3.256401462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:47:36.73692848 +0000 UTC m=+723.822483685" watchObservedRunningTime="2026-07-06 20:47:38.256401462 +0000 UTC m=+725.341956659" Jul 06 20:47:38.374100 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.374075 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-kube-rbac-proxy-sar-config\") pod \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\" (UID: \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\") " Jul 06 20:47:38.374256 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.374137 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqstn\" (UniqueName: \"kubernetes.io/projected/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-kube-api-access-cqstn\") pod \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\" (UID: \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\") " Jul 06 20:47:38.374256 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.374160 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-proxy-tls\") pod \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\" (UID: \"601c9d0f-a9df-4b87-942a-173b3ac9d8b2\") " Jul 06 20:47:38.374475 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.374452 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "601c9d0f-a9df-4b87-942a-173b3ac9d8b2" (UID: "601c9d0f-a9df-4b87-942a-173b3ac9d8b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:47:38.376434 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.376408 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "601c9d0f-a9df-4b87-942a-173b3ac9d8b2" (UID: "601c9d0f-a9df-4b87-942a-173b3ac9d8b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:47:38.376434 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.376420 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-kube-api-access-cqstn" (OuterVolumeSpecName: "kube-api-access-cqstn") pod "601c9d0f-a9df-4b87-942a-173b3ac9d8b2" (UID: "601c9d0f-a9df-4b87-942a-173b3ac9d8b2"). InnerVolumeSpecName "kube-api-access-cqstn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:47:38.474420 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.474393 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:47:38.474625 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.474607 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:47:38.474684 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.474629 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqstn\" (UniqueName: \"kubernetes.io/projected/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-kube-api-access-cqstn\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:47:38.474684 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.474638 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601c9d0f-a9df-4b87-942a-173b3ac9d8b2-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:47:38.575795 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.575746 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-kube-rbac-proxy-sar-config\") pod \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\" (UID: \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\") " Jul 06 20:47:38.576002 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.575870 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-proxy-tls\") pod \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\" (UID: \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\") " Jul 06 20:47:38.576002 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.575916 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4jst\" (UniqueName: \"kubernetes.io/projected/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-kube-api-access-v4jst\") pod \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\" (UID: \"ae9160a2-4983-46e5-aa8c-c2ae98c0af05\") " Jul 06 20:47:38.576166 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.576146 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae9160a2-4983-46e5-aa8c-c2ae98c0af05" (UID: "ae9160a2-4983-46e5-aa8c-c2ae98c0af05"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:47:38.577862 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.577840 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae9160a2-4983-46e5-aa8c-c2ae98c0af05" (UID: "ae9160a2-4983-46e5-aa8c-c2ae98c0af05"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:47:38.577991 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.577970 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-kube-api-access-v4jst" (OuterVolumeSpecName: "kube-api-access-v4jst") pod "ae9160a2-4983-46e5-aa8c-c2ae98c0af05" (UID: "ae9160a2-4983-46e5-aa8c-c2ae98c0af05"). InnerVolumeSpecName "kube-api-access-v4jst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:47:38.676743 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.676708 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:47:38.676743 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.676739 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:47:38.676968 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.676752 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4jst\" (UniqueName: \"kubernetes.io/projected/ae9160a2-4983-46e5-aa8c-c2ae98c0af05-kube-api-access-v4jst\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:47:38.711506 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.711469 2582 generic.go:356] "Generic (PLEG): container finished" podID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerID="c9770be4d2de2f9f58ac7ea52ba55dac7483c4fd8d3a9d32fd257d4c01179d74" exitCode=0 Jul 06 20:47:38.711976 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.711521 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" Jul 06 20:47:38.711976 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.711555 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" event={"ID":"ae9160a2-4983-46e5-aa8c-c2ae98c0af05","Type":"ContainerDied","Data":"c9770be4d2de2f9f58ac7ea52ba55dac7483c4fd8d3a9d32fd257d4c01179d74"} Jul 06 20:47:38.711976 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.711630 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd" event={"ID":"ae9160a2-4983-46e5-aa8c-c2ae98c0af05","Type":"ContainerDied","Data":"48dde3772c5cc070ea440a6d720d20e4f06780e1c54b63b4601a188d6cba0e3d"} Jul 06 20:47:38.711976 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.711673 2582 scope.go:117] "RemoveContainer" containerID="b732d08c2555736478fa0930ea851b19dc6aec88f4e363277ceced57a44911af" Jul 06 20:47:38.713367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.713344 2582 generic.go:356] "Generic (PLEG): container finished" podID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerID="c80d12666aa62754540ba588ed2ef9d1829d511e4cee75bc5ee75ca12faa823c" exitCode=0 Jul 06 20:47:38.713481 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.713409 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" event={"ID":"601c9d0f-a9df-4b87-942a-173b3ac9d8b2","Type":"ContainerDied","Data":"c80d12666aa62754540ba588ed2ef9d1829d511e4cee75bc5ee75ca12faa823c"} Jul 06 20:47:38.713481 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.713467 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" event={"ID":"601c9d0f-a9df-4b87-942a-173b3ac9d8b2","Type":"ContainerDied","Data":"a49fe4214ea0fcc0894d8e8a25ec30005bf61dce9c63cfe41d48e57b468adc8d"} Jul 06 20:47:38.713574 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.713415 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k" Jul 06 20:47:38.720560 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.720514 2582 scope.go:117] "RemoveContainer" containerID="c9770be4d2de2f9f58ac7ea52ba55dac7483c4fd8d3a9d32fd257d4c01179d74" Jul 06 20:47:38.729399 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.729377 2582 scope.go:117] "RemoveContainer" containerID="b732d08c2555736478fa0930ea851b19dc6aec88f4e363277ceced57a44911af" Jul 06 20:47:38.729657 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:47:38.729638 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b732d08c2555736478fa0930ea851b19dc6aec88f4e363277ceced57a44911af\": container with ID starting with b732d08c2555736478fa0930ea851b19dc6aec88f4e363277ceced57a44911af not found: ID does not exist" containerID="b732d08c2555736478fa0930ea851b19dc6aec88f4e363277ceced57a44911af" Jul 06 20:47:38.729711 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.729665 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b732d08c2555736478fa0930ea851b19dc6aec88f4e363277ceced57a44911af"} err="failed to get container status \"b732d08c2555736478fa0930ea851b19dc6aec88f4e363277ceced57a44911af\": rpc error: code = NotFound desc = could not find container \"b732d08c2555736478fa0930ea851b19dc6aec88f4e363277ceced57a44911af\": container with ID starting with b732d08c2555736478fa0930ea851b19dc6aec88f4e363277ceced57a44911af not found: ID does not exist" Jul 06 20:47:38.729711 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.729683 2582 scope.go:117] "RemoveContainer" containerID="c9770be4d2de2f9f58ac7ea52ba55dac7483c4fd8d3a9d32fd257d4c01179d74" Jul 06 20:47:38.729953 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:47:38.729936 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9770be4d2de2f9f58ac7ea52ba55dac7483c4fd8d3a9d32fd257d4c01179d74\": container with ID starting with c9770be4d2de2f9f58ac7ea52ba55dac7483c4fd8d3a9d32fd257d4c01179d74 not found: ID does not exist" containerID="c9770be4d2de2f9f58ac7ea52ba55dac7483c4fd8d3a9d32fd257d4c01179d74" Jul 06 20:47:38.730003 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.729957 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9770be4d2de2f9f58ac7ea52ba55dac7483c4fd8d3a9d32fd257d4c01179d74"} err="failed to get container status \"c9770be4d2de2f9f58ac7ea52ba55dac7483c4fd8d3a9d32fd257d4c01179d74\": rpc error: code = NotFound desc = could not find container \"c9770be4d2de2f9f58ac7ea52ba55dac7483c4fd8d3a9d32fd257d4c01179d74\": container with ID starting with c9770be4d2de2f9f58ac7ea52ba55dac7483c4fd8d3a9d32fd257d4c01179d74 not found: ID does not exist" Jul 06 20:47:38.730003 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.729971 2582 scope.go:117] "RemoveContainer" containerID="756d03b3971a2a0545736b17e37fb16259853349bac4394908c4044fb24b4413" Jul 06 20:47:38.737175 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.737153 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd"] Jul 06 20:47:38.738913 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.738888 2582 scope.go:117] "RemoveContainer" containerID="c80d12666aa62754540ba588ed2ef9d1829d511e4cee75bc5ee75ca12faa823c" Jul 06 20:47:38.744135 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.744117 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-85420-predictor-856b7d987-9rsjd"] Jul 06 20:47:38.748191 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.748173 2582 scope.go:117] "RemoveContainer" containerID="756d03b3971a2a0545736b17e37fb16259853349bac4394908c4044fb24b4413" Jul 06 20:47:38.748448 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:47:38.748427 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756d03b3971a2a0545736b17e37fb16259853349bac4394908c4044fb24b4413\": container with ID starting with 756d03b3971a2a0545736b17e37fb16259853349bac4394908c4044fb24b4413 not found: ID does not exist" containerID="756d03b3971a2a0545736b17e37fb16259853349bac4394908c4044fb24b4413" Jul 06 20:47:38.748537 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.748453 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756d03b3971a2a0545736b17e37fb16259853349bac4394908c4044fb24b4413"} err="failed to get container status \"756d03b3971a2a0545736b17e37fb16259853349bac4394908c4044fb24b4413\": rpc error: code = NotFound desc = could not find container \"756d03b3971a2a0545736b17e37fb16259853349bac4394908c4044fb24b4413\": container with ID starting with 756d03b3971a2a0545736b17e37fb16259853349bac4394908c4044fb24b4413 not found: ID does not exist" Jul 06 20:47:38.748537 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.748476 2582 scope.go:117] "RemoveContainer" containerID="c80d12666aa62754540ba588ed2ef9d1829d511e4cee75bc5ee75ca12faa823c" Jul 06 20:47:38.748739 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:47:38.748723 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80d12666aa62754540ba588ed2ef9d1829d511e4cee75bc5ee75ca12faa823c\": container with ID starting with c80d12666aa62754540ba588ed2ef9d1829d511e4cee75bc5ee75ca12faa823c not found: ID does not exist" containerID="c80d12666aa62754540ba588ed2ef9d1829d511e4cee75bc5ee75ca12faa823c" Jul 06 20:47:38.748819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.748748 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80d12666aa62754540ba588ed2ef9d1829d511e4cee75bc5ee75ca12faa823c"} err="failed to get container status \"c80d12666aa62754540ba588ed2ef9d1829d511e4cee75bc5ee75ca12faa823c\": rpc error: code = NotFound desc = could not find container \"c80d12666aa62754540ba588ed2ef9d1829d511e4cee75bc5ee75ca12faa823c\": container with ID starting with c80d12666aa62754540ba588ed2ef9d1829d511e4cee75bc5ee75ca12faa823c not found: ID does not exist" Jul 06 20:47:38.752655 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:47:38.752628 2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9160a2_4983_46e5_aa8c_c2ae98c0af05.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9160a2_4983_46e5_aa8c_c2ae98c0af05.slice/crio-48dde3772c5cc070ea440a6d720d20e4f06780e1c54b63b4601a188d6cba0e3d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601c9d0f_a9df_4b87_942a_173b3ac9d8b2.slice/crio-a49fe4214ea0fcc0894d8e8a25ec30005bf61dce9c63cfe41d48e57b468adc8d\": RecentStats: unable to find data in memory cache]" Jul 06 20:47:38.759456 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.759435 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k"] Jul 06 20:47:38.764347 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:38.764326 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-85420-predictor-8c694c98b-cnn9k"] Jul 06 20:47:39.544992 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:39.544952 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" path="/var/lib/kubelet/pods/601c9d0f-a9df-4b87-942a-173b3ac9d8b2/volumes" Jul 06 20:47:39.545398 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:39.545385 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" path="/var/lib/kubelet/pods/ae9160a2-4983-46e5-aa8c-c2ae98c0af05/volumes" Jul 06 20:47:41.705371 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:41.705342 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:47:41.705867 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:41.705748 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Jul 06 20:47:42.709710 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:42.709684 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:47:42.710277 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:42.710248 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 06 20:47:51.706286 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:51.706248 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Jul 06 20:47:52.710843 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:47:52.710808 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 06 20:48:01.706155 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:01.706114 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Jul 06 20:48:02.710860 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:02.710818 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 06 20:48:11.706810 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:11.706753 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Jul 06 20:48:12.710290 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:12.710249 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 06 20:48:14.908096 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.908066 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq"] Jul 06 20:48:14.908434 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.908400 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kube-rbac-proxy" Jul 06 20:48:14.908434 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.908412 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kube-rbac-proxy" Jul 06 20:48:14.908434 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.908430 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kserve-container" Jul 06 20:48:14.908434 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.908436 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kserve-container" Jul 06 20:48:14.908563 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.908442 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kube-rbac-proxy" Jul 06 20:48:14.908563 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.908448 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kube-rbac-proxy" Jul 06 20:48:14.908563 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.908454 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kserve-container" Jul 06 20:48:14.908563 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.908459 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kserve-container" Jul 06 20:48:14.908563 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.908513 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kube-rbac-proxy" Jul 06 20:48:14.908563 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.908521 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kube-rbac-proxy" Jul 06 20:48:14.908563 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.908530 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae9160a2-4983-46e5-aa8c-c2ae98c0af05" containerName="kserve-container" Jul 06 20:48:14.908563 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.908536 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="601c9d0f-a9df-4b87-942a-173b3ac9d8b2" containerName="kserve-container" Jul 06 20:48:14.911824 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.911800 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:14.915100 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.915082 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dfad7-predictor-serving-cert\"" Jul 06 20:48:14.915188 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.915083 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dfad7-kube-rbac-proxy-sar-config\"" Jul 06 20:48:14.924550 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.924526 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq"] Jul 06 20:48:14.964360 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.964334 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t76q6\" (UniqueName: \"kubernetes.io/projected/18578650-14bd-4276-989b-41074feddf54-kube-api-access-t76q6\") pod \"success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq\" (UID: \"18578650-14bd-4276-989b-41074feddf54\") " pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:14.964468 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.964368 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18578650-14bd-4276-989b-41074feddf54-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq\" (UID: \"18578650-14bd-4276-989b-41074feddf54\") " pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:14.964550 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:14.964526 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18578650-14bd-4276-989b-41074feddf54-proxy-tls\") pod \"success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq\" (UID: \"18578650-14bd-4276-989b-41074feddf54\") " pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:15.023468 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.023443 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq"] Jul 06 20:48:15.027047 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.027027 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:15.029982 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.029958 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dfad7-kube-rbac-proxy-sar-config\"" Jul 06 20:48:15.030224 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.030207 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dfad7-predictor-serving-cert\"" Jul 06 20:48:15.046692 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.046666 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq"] Jul 06 20:48:15.065417 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.065384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18578650-14bd-4276-989b-41074feddf54-proxy-tls\") pod \"success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq\" (UID: \"18578650-14bd-4276-989b-41074feddf54\") " pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:15.065558 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.065423 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t76q6\" (UniqueName: \"kubernetes.io/projected/18578650-14bd-4276-989b-41074feddf54-kube-api-access-t76q6\") pod \"success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq\" (UID: \"18578650-14bd-4276-989b-41074feddf54\") " pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:15.065558 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.065447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18578650-14bd-4276-989b-41074feddf54-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq\" (UID: \"18578650-14bd-4276-989b-41074feddf54\") " pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:15.065558 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.065471 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce90d61a-992b-4ab1-81e4-daeed0701f53-proxy-tls\") pod \"error-404-isvc-dfad7-predictor-9f88c844f-hddzq\" (UID: \"ce90d61a-992b-4ab1-81e4-daeed0701f53\") " pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:15.065558 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.065544 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce90d61a-992b-4ab1-81e4-daeed0701f53-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dfad7-predictor-9f88c844f-hddzq\" (UID: \"ce90d61a-992b-4ab1-81e4-daeed0701f53\") " pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:15.065839 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.065587 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpqtm\" (UniqueName: \"kubernetes.io/projected/ce90d61a-992b-4ab1-81e4-daeed0701f53-kube-api-access-xpqtm\") pod \"error-404-isvc-dfad7-predictor-9f88c844f-hddzq\" (UID: \"ce90d61a-992b-4ab1-81e4-daeed0701f53\") " pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:15.066267 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.066236 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18578650-14bd-4276-989b-41074feddf54-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq\" (UID: \"18578650-14bd-4276-989b-41074feddf54\") " pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:15.068626 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.068597 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18578650-14bd-4276-989b-41074feddf54-proxy-tls\") pod \"success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq\" (UID: \"18578650-14bd-4276-989b-41074feddf54\") " pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:15.075828 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.075803 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t76q6\" (UniqueName: \"kubernetes.io/projected/18578650-14bd-4276-989b-41074feddf54-kube-api-access-t76q6\") pod \"success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq\" (UID: \"18578650-14bd-4276-989b-41074feddf54\") " pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:15.166703 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.166620 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpqtm\" (UniqueName: \"kubernetes.io/projected/ce90d61a-992b-4ab1-81e4-daeed0701f53-kube-api-access-xpqtm\") pod \"error-404-isvc-dfad7-predictor-9f88c844f-hddzq\" (UID: \"ce90d61a-992b-4ab1-81e4-daeed0701f53\") " pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:15.166874 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.166722 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce90d61a-992b-4ab1-81e4-daeed0701f53-proxy-tls\") pod \"error-404-isvc-dfad7-predictor-9f88c844f-hddzq\" (UID: \"ce90d61a-992b-4ab1-81e4-daeed0701f53\") " pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:15.166874 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.166744 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce90d61a-992b-4ab1-81e4-daeed0701f53-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dfad7-predictor-9f88c844f-hddzq\" (UID: \"ce90d61a-992b-4ab1-81e4-daeed0701f53\") " pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:15.166990 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:48:15.166880 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-serving-cert: secret "error-404-isvc-dfad7-predictor-serving-cert" not found Jul 06 20:48:15.166990 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:48:15.166948 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce90d61a-992b-4ab1-81e4-daeed0701f53-proxy-tls podName:ce90d61a-992b-4ab1-81e4-daeed0701f53 nodeName:}" failed. No retries permitted until 2026-07-06 20:48:15.666929616 +0000 UTC m=+762.752484791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ce90d61a-992b-4ab1-81e4-daeed0701f53-proxy-tls") pod "error-404-isvc-dfad7-predictor-9f88c844f-hddzq" (UID: "ce90d61a-992b-4ab1-81e4-daeed0701f53") : secret "error-404-isvc-dfad7-predictor-serving-cert" not found Jul 06 20:48:15.167417 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.167399 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce90d61a-992b-4ab1-81e4-daeed0701f53-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dfad7-predictor-9f88c844f-hddzq\" (UID: \"ce90d61a-992b-4ab1-81e4-daeed0701f53\") " pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:15.179548 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.179519 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpqtm\" (UniqueName: \"kubernetes.io/projected/ce90d61a-992b-4ab1-81e4-daeed0701f53-kube-api-access-xpqtm\") pod \"error-404-isvc-dfad7-predictor-9f88c844f-hddzq\" (UID: \"ce90d61a-992b-4ab1-81e4-daeed0701f53\") " pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:15.227242 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.227214 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:15.364835 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.364813 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq"] Jul 06 20:48:15.366987 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:48:15.366961 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18578650_14bd_4276_989b_41074feddf54.slice/crio-1aa0a7edad83e413d15a3b685be06f52d0181415c944b31dc31ce3e535bd59e7 WatchSource:0}: Error finding container 1aa0a7edad83e413d15a3b685be06f52d0181415c944b31dc31ce3e535bd59e7: Status 404 returned error can't find the container with id 1aa0a7edad83e413d15a3b685be06f52d0181415c944b31dc31ce3e535bd59e7 Jul 06 20:48:15.671820 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.671784 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce90d61a-992b-4ab1-81e4-daeed0701f53-proxy-tls\") pod \"error-404-isvc-dfad7-predictor-9f88c844f-hddzq\" (UID: \"ce90d61a-992b-4ab1-81e4-daeed0701f53\") " pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:15.674118 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.674093 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce90d61a-992b-4ab1-81e4-daeed0701f53-proxy-tls\") pod \"error-404-isvc-dfad7-predictor-9f88c844f-hddzq\" (UID: \"ce90d61a-992b-4ab1-81e4-daeed0701f53\") " pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:15.888009 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.887978 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" event={"ID":"18578650-14bd-4276-989b-41074feddf54","Type":"ContainerStarted","Data":"5d5933fdf8107341c87866f9502aa83b85a6e0325d484c697569a8f9174a79b4"} Jul 06 20:48:15.888166 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.888031 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" event={"ID":"18578650-14bd-4276-989b-41074feddf54","Type":"ContainerStarted","Data":"89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b"} Jul 06 20:48:15.888166 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.888051 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" event={"ID":"18578650-14bd-4276-989b-41074feddf54","Type":"ContainerStarted","Data":"1aa0a7edad83e413d15a3b685be06f52d0181415c944b31dc31ce3e535bd59e7"} Jul 06 20:48:15.888166 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.888077 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:15.888166 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.888135 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:15.889441 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.889416 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Jul 06 20:48:15.911873 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.911829 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" podStartSLOduration=1.911816914 podStartE2EDuration="1.911816914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:15.910661976 +0000 UTC m=+762.996217161" watchObservedRunningTime="2026-07-06 20:48:15.911816914 +0000 UTC m=+762.997372107" Jul 06 20:48:15.937070 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:15.937045 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:16.077704 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:16.077673 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq"] Jul 06 20:48:16.080445 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:48:16.080397 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce90d61a_992b_4ab1_81e4_daeed0701f53.slice/crio-67f28f515209abe606012447fb556ae6a782338083fe91ae94016167e3210e67 WatchSource:0}: Error finding container 67f28f515209abe606012447fb556ae6a782338083fe91ae94016167e3210e67: Status 404 returned error can't find the container with id 67f28f515209abe606012447fb556ae6a782338083fe91ae94016167e3210e67 Jul 06 20:48:16.894776 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:16.894727 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" event={"ID":"ce90d61a-992b-4ab1-81e4-daeed0701f53","Type":"ContainerStarted","Data":"513a1f7d6890580eb71f71ec71255c22563c9c9efae81edec2e0fcbdd2bb1762"} Jul 06 20:48:16.894953 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:16.894809 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" event={"ID":"ce90d61a-992b-4ab1-81e4-daeed0701f53","Type":"ContainerStarted","Data":"bd57a626e813bb9f91fda0b463007283d307eee53cdb79b4b561677dfe397e10"} Jul 06 20:48:16.894953 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:16.894839 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" event={"ID":"ce90d61a-992b-4ab1-81e4-daeed0701f53","Type":"ContainerStarted","Data":"67f28f515209abe606012447fb556ae6a782338083fe91ae94016167e3210e67"} Jul 06 20:48:16.895057 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:16.894999 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Jul 06 20:48:16.895108 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:16.895082 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:16.928257 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:16.928213 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" podStartSLOduration=1.928200804 podStartE2EDuration="1.928200804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:16.926404963 +0000 UTC m=+764.011960157" watchObservedRunningTime="2026-07-06 20:48:16.928200804 +0000 UTC m=+764.013756062" Jul 06 20:48:17.899779 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:17.899735 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:17.900806 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:17.900755 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 06 20:48:18.903871 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:18.903830 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 06 20:48:21.706712 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:21.706684 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:48:21.900157 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:21.900129 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:48:21.900799 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:21.900773 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Jul 06 20:48:22.712035 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:22.712001 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:48:23.908722 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:23.908691 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:48:23.909137 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:23.909111 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 06 20:48:31.900897 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:31.900842 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Jul 06 20:48:33.909137 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:33.909104 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 06 20:48:41.900799 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:41.900745 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Jul 06 20:48:43.910088 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:43.910048 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 06 20:48:51.901349 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:51.901306 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Jul 06 20:48:53.909402 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:48:53.909363 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" Jul 06 20:49:01.902571 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:49:01.902487 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:49:03.909922 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:49:03.909892 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:50:33.390283 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:50:33.390255 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 20:50:33.393044 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:50:33.393021 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 20:55:33.422090 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:55:33.422057 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 20:55:33.431032 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:55:33.431009 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 20:56:49.987912 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:49.987871 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf"] Jul 06 20:56:49.991872 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:49.991855 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2"] Jul 06 20:56:49.991990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:49.991972 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:49.992079 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:49.992057 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kserve-container" containerID="cri-o://fdbdac8e2a5dfd934a389d00c54256f2ee31ae0e63941405e3a8ac75a479d3cf" gracePeriod=30 Jul 06 20:56:49.992079 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:49.992071 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kube-rbac-proxy" containerID="cri-o://9d983b28894a59129f6c4baecec5c514f8e729e0d666dab23f74363469909f8c" gracePeriod=30 Jul 06 20:56:49.994920 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:49.994901 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-305f4-kube-rbac-proxy-sar-config\"" Jul 06 20:56:49.995035 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:49.994903 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-305f4-predictor-serving-cert\"" Jul 06 20:56:50.004639 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.004609 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf"] Jul 06 20:56:50.077430 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.077396 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2"] Jul 06 20:56:50.080921 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.080897 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:50.083639 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.083618 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-305f4-kube-rbac-proxy-sar-config\"" Jul 06 20:56:50.083901 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.083884 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-305f4-predictor-serving-cert\"" Jul 06 20:56:50.086302 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.086281 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-proxy-tls\") pod \"success-200-isvc-305f4-predictor-c7fb94cc6-66jqf\" (UID: \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\") " pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:50.086388 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.086332 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-305f4-predictor-c7fb94cc6-66jqf\" (UID: \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\") " pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:50.086453 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.086442 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs2ps\" (UniqueName: \"kubernetes.io/projected/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-kube-api-access-cs2ps\") pod \"success-200-isvc-305f4-predictor-c7fb94cc6-66jqf\" (UID: \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\") " pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:50.091724 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.091699 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2"] Jul 06 20:56:50.123942 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.123913 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c"] Jul 06 20:56:50.124215 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.124171 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kserve-container" containerID="cri-o://88c9b2abcb11c8ce1ffdc192b2c23f4a344a4def752699543c0e2d73cb5b5b59" gracePeriod=30 Jul 06 20:56:50.124445 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.124245 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kube-rbac-proxy" containerID="cri-o://df8a12397de10060af8fe855f43153cb1ef6d9562d60c94cf9ea6e136bfdb6e5" gracePeriod=30 Jul 06 20:56:50.186962 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.186928 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv6hc\" (UniqueName: \"kubernetes.io/projected/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-kube-api-access-nv6hc\") pod \"error-404-isvc-305f4-predictor-5d5987d4f-pz2z2\" (UID: \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\") " pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:50.187120 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.186976 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-305f4-predictor-5d5987d4f-pz2z2\" (UID: \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\") " pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:50.187120 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.187000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs2ps\" (UniqueName: \"kubernetes.io/projected/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-kube-api-access-cs2ps\") pod \"success-200-isvc-305f4-predictor-c7fb94cc6-66jqf\" (UID: \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\") " pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:50.187120 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.187034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-proxy-tls\") pod \"success-200-isvc-305f4-predictor-c7fb94cc6-66jqf\" (UID: \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\") " pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:50.187120 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.187050 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-proxy-tls\") pod \"error-404-isvc-305f4-predictor-5d5987d4f-pz2z2\" (UID: \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\") " pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:50.187120 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.187083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-305f4-predictor-c7fb94cc6-66jqf\" (UID: \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\") " pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:50.187280 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:56:50.187143 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-305f4-predictor-serving-cert: secret "success-200-isvc-305f4-predictor-serving-cert" not found Jul 06 20:56:50.187280 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:56:50.187204 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-proxy-tls podName:d4b9da3e-0e8f-4fd4-9550-82b899e84d89 nodeName:}" failed. No retries permitted until 2026-07-06 20:56:50.687184115 +0000 UTC m=+1277.772739297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-proxy-tls") pod "success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" (UID: "d4b9da3e-0e8f-4fd4-9550-82b899e84d89") : secret "success-200-isvc-305f4-predictor-serving-cert" not found Jul 06 20:56:50.187655 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.187638 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-305f4-predictor-c7fb94cc6-66jqf\" (UID: \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\") " pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:50.197945 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.197923 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs2ps\" (UniqueName: \"kubernetes.io/projected/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-kube-api-access-cs2ps\") pod \"success-200-isvc-305f4-predictor-c7fb94cc6-66jqf\" (UID: \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\") " pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:50.212471 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.212444 2582 generic.go:356] "Generic (PLEG): container finished" podID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerID="9d983b28894a59129f6c4baecec5c514f8e729e0d666dab23f74363469909f8c" exitCode=2 Jul 06 20:56:50.212583 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.212513 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" event={"ID":"ff6dd3d9-a337-45c3-a51a-218105f7ff8e","Type":"ContainerDied","Data":"9d983b28894a59129f6c4baecec5c514f8e729e0d666dab23f74363469909f8c"} Jul 06 20:56:50.287494 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.287460 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-305f4-predictor-5d5987d4f-pz2z2\" (UID: \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\") " pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:50.287645 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.287525 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-proxy-tls\") pod \"error-404-isvc-305f4-predictor-5d5987d4f-pz2z2\" (UID: \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\") " pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:50.287645 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.287588 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6hc\" (UniqueName: \"kubernetes.io/projected/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-kube-api-access-nv6hc\") pod \"error-404-isvc-305f4-predictor-5d5987d4f-pz2z2\" (UID: \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\") " pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:50.287732 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:56:50.287696 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-305f4-predictor-serving-cert: secret "error-404-isvc-305f4-predictor-serving-cert" not found Jul 06 20:56:50.287809 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:56:50.287782 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-proxy-tls podName:5d6f37bd-f266-4ce2-b814-c84d82b52b2a nodeName:}" failed. No retries permitted until 2026-07-06 20:56:50.787741335 +0000 UTC m=+1277.873296507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-proxy-tls") pod "error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" (UID: "5d6f37bd-f266-4ce2-b814-c84d82b52b2a") : secret "error-404-isvc-305f4-predictor-serving-cert" not found Jul 06 20:56:50.288216 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.288198 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-305f4-predictor-5d5987d4f-pz2z2\" (UID: \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\") " pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:50.297057 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.297034 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv6hc\" (UniqueName: \"kubernetes.io/projected/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-kube-api-access-nv6hc\") pod \"error-404-isvc-305f4-predictor-5d5987d4f-pz2z2\" (UID: \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\") " pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:50.691604 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.691567 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-proxy-tls\") pod \"success-200-isvc-305f4-predictor-c7fb94cc6-66jqf\" (UID: \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\") " pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:50.693949 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.693921 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-proxy-tls\") pod \"success-200-isvc-305f4-predictor-c7fb94cc6-66jqf\" (UID: \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\") " pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:50.792239 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.792192 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-proxy-tls\") pod \"error-404-isvc-305f4-predictor-5d5987d4f-pz2z2\" (UID: \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\") " pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:50.794649 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.794624 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-proxy-tls\") pod \"error-404-isvc-305f4-predictor-5d5987d4f-pz2z2\" (UID: \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\") " pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:50.912559 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.912526 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:50.991922 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:50.991885 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:51.054976 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.054888 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf"] Jul 06 20:56:51.058433 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:56:51.058383 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b9da3e_0e8f_4fd4_9550_82b899e84d89.slice/crio-b8dd057c02b5ecb5a16186f2c366deae98a66889676d97703b4d7bee01ff80eb WatchSource:0}: Error finding container b8dd057c02b5ecb5a16186f2c366deae98a66889676d97703b4d7bee01ff80eb: Status 404 returned error can't find the container with id b8dd057c02b5ecb5a16186f2c366deae98a66889676d97703b4d7bee01ff80eb Jul 06 20:56:51.147943 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.147916 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2"] Jul 06 20:56:51.149785 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:56:51.149744 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6f37bd_f266_4ce2_b814_c84d82b52b2a.slice/crio-301a46dcd6b0fa5ae92e1bf98c540fbd81ceaa8d0f36934f1874d61e73abc6ec WatchSource:0}: Error finding container 301a46dcd6b0fa5ae92e1bf98c540fbd81ceaa8d0f36934f1874d61e73abc6ec: Status 404 returned error can't find the container with id 301a46dcd6b0fa5ae92e1bf98c540fbd81ceaa8d0f36934f1874d61e73abc6ec Jul 06 20:56:51.219072 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.219012 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" event={"ID":"d4b9da3e-0e8f-4fd4-9550-82b899e84d89","Type":"ContainerStarted","Data":"897b1a2f03f20cb6803bb5533189884cbdcf6aa2eea3c4af43a85d52e7f7d31b"} Jul 06 20:56:51.219183 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.219073 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" event={"ID":"d4b9da3e-0e8f-4fd4-9550-82b899e84d89","Type":"ContainerStarted","Data":"ed34c900b5b8c2b29aff2bbb00998672c38c40711e7d3749f27dd4aec1bd35cc"} Jul 06 20:56:51.219183 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.219097 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" event={"ID":"d4b9da3e-0e8f-4fd4-9550-82b899e84d89","Type":"ContainerStarted","Data":"b8dd057c02b5ecb5a16186f2c366deae98a66889676d97703b4d7bee01ff80eb"} Jul 06 20:56:51.219183 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.219130 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:51.219335 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.219226 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:51.220848 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.220825 2582 generic.go:356] "Generic (PLEG): container finished" podID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerID="df8a12397de10060af8fe855f43153cb1ef6d9562d60c94cf9ea6e136bfdb6e5" exitCode=2 Jul 06 20:56:51.220947 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.220841 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 06 20:56:51.220947 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.220904 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" event={"ID":"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3","Type":"ContainerDied","Data":"df8a12397de10060af8fe855f43153cb1ef6d9562d60c94cf9ea6e136bfdb6e5"} Jul 06 20:56:51.222251 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.222234 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" event={"ID":"5d6f37bd-f266-4ce2-b814-c84d82b52b2a","Type":"ContainerStarted","Data":"841b4de1ae6e309fb10be1bc8eb97294546b0e653072b930ff53f8a78743e788"} Jul 06 20:56:51.222326 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.222280 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" event={"ID":"5d6f37bd-f266-4ce2-b814-c84d82b52b2a","Type":"ContainerStarted","Data":"301a46dcd6b0fa5ae92e1bf98c540fbd81ceaa8d0f36934f1874d61e73abc6ec"} Jul 06 20:56:51.701286 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.701234 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused" Jul 06 20:56:51.706690 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:51.706655 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Jul 06 20:56:52.230847 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:52.230808 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" event={"ID":"5d6f37bd-f266-4ce2-b814-c84d82b52b2a","Type":"ContainerStarted","Data":"ab0cb4f30ce8c923fd1117eb82cb463cbe8743b852f418964a51f719a85de243"} Jul 06 20:56:52.231301 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:52.230930 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:52.231301 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:52.230971 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:52.235421 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:52.235380 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 06 20:56:52.235609 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:52.235386 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" Jul 06 20:56:52.257164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:52.257109 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" podStartSLOduration=2.257094004 podStartE2EDuration="2.257094004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:56:52.255102759 +0000 UTC m=+1279.340657954" watchObservedRunningTime="2026-07-06 20:56:52.257094004 +0000 UTC m=+1279.342649197" Jul 06 20:56:52.257365 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:52.257300 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" podStartSLOduration=3.257293854 podStartE2EDuration="3.257293854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:56:51.243784887 +0000 UTC m=+1278.329340074" watchObservedRunningTime="2026-07-06 20:56:52.257293854 +0000 UTC m=+1279.342849050" Jul 06 20:56:52.706250 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:52.706203 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused" Jul 06 20:56:52.711096 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:52.711063 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" Jul 06 20:56:53.233897 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.233857 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" Jul 06 20:56:53.614774 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.614739 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:56:53.716597 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.716563 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx5gf\" (UniqueName: \"kubernetes.io/projected/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-kube-api-access-jx5gf\") pod \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\" (UID: \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\") " Jul 06 20:56:53.716778 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.716611 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-kube-rbac-proxy-sar-config\") pod \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\" (UID: \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\") " Jul 06 20:56:53.716778 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.716742 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-proxy-tls\") pod \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\" (UID: \"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3\") " Jul 06 20:56:53.717022 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.716993 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" (UID: "40e7d7b2-3e68-46e5-ab5f-80aed349e5f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:56:53.720323 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.720292 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" (UID: "40e7d7b2-3e68-46e5-ab5f-80aed349e5f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:56:53.720421 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.720392 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-kube-api-access-jx5gf" (OuterVolumeSpecName: "kube-api-access-jx5gf") pod "40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" (UID: "40e7d7b2-3e68-46e5-ab5f-80aed349e5f3"). InnerVolumeSpecName "kube-api-access-jx5gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:56:53.743567 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.743536 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:56:53.818028 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.817945 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptlhf\" (UniqueName: \"kubernetes.io/projected/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-kube-api-access-ptlhf\") pod \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\" (UID: \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\") " Jul 06 20:56:53.818028 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.818001 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-proxy-tls\") pod \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\" (UID: \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\") " Jul 06 20:56:53.818207 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.818039 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-kube-rbac-proxy-sar-config\") pod \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\" (UID: \"ff6dd3d9-a337-45c3-a51a-218105f7ff8e\") " Jul 06 20:56:53.818243 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.818207 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx5gf\" (UniqueName: \"kubernetes.io/projected/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-kube-api-access-jx5gf\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:56:53.818243 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.818219 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:56:53.818243 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.818228 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:56:53.818485 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.818461 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff6dd3d9-a337-45c3-a51a-218105f7ff8e" (UID: "ff6dd3d9-a337-45c3-a51a-218105f7ff8e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:56:53.819948 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.819927 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-kube-api-access-ptlhf" (OuterVolumeSpecName: "kube-api-access-ptlhf") pod "ff6dd3d9-a337-45c3-a51a-218105f7ff8e" (UID: "ff6dd3d9-a337-45c3-a51a-218105f7ff8e"). InnerVolumeSpecName "kube-api-access-ptlhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:56:53.820023 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.820004 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff6dd3d9-a337-45c3-a51a-218105f7ff8e" (UID: "ff6dd3d9-a337-45c3-a51a-218105f7ff8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:56:53.918612 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.918578 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptlhf\" (UniqueName: \"kubernetes.io/projected/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-kube-api-access-ptlhf\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:56:53.918612 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.918603 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:56:53.918612 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:53.918614 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6dd3d9-a337-45c3-a51a-218105f7ff8e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:56:54.239519 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.239480 2582 generic.go:356] "Generic (PLEG): container finished" podID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerID="fdbdac8e2a5dfd934a389d00c54256f2ee31ae0e63941405e3a8ac75a479d3cf" exitCode=0 Jul 06 20:56:54.240007 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.239542 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" Jul 06 20:56:54.240007 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.239563 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" event={"ID":"ff6dd3d9-a337-45c3-a51a-218105f7ff8e","Type":"ContainerDied","Data":"fdbdac8e2a5dfd934a389d00c54256f2ee31ae0e63941405e3a8ac75a479d3cf"} Jul 06 20:56:54.240007 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.239624 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2" event={"ID":"ff6dd3d9-a337-45c3-a51a-218105f7ff8e","Type":"ContainerDied","Data":"3c8ef28d12f8664ec4bf42b858ebf828dd072c1dac7fa0a5fba2743a72277b0f"} Jul 06 20:56:54.240007 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.239658 2582 scope.go:117] "RemoveContainer" containerID="9d983b28894a59129f6c4baecec5c514f8e729e0d666dab23f74363469909f8c" Jul 06 20:56:54.241512 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.241480 2582 generic.go:356] "Generic (PLEG): container finished" podID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerID="88c9b2abcb11c8ce1ffdc192b2c23f4a344a4def752699543c0e2d73cb5b5b59" exitCode=0 Jul 06 20:56:54.241608 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.241539 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" Jul 06 20:56:54.241608 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.241554 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" event={"ID":"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3","Type":"ContainerDied","Data":"88c9b2abcb11c8ce1ffdc192b2c23f4a344a4def752699543c0e2d73cb5b5b59"} Jul 06 20:56:54.241608 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.241603 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c" event={"ID":"40e7d7b2-3e68-46e5-ab5f-80aed349e5f3","Type":"ContainerDied","Data":"32e18378ee1fe7d4a1043ddc1d7232d743d78a1786692441ef556dbc313f36c0"} Jul 06 20:56:54.249567 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.249549 2582 scope.go:117] "RemoveContainer" containerID="fdbdac8e2a5dfd934a389d00c54256f2ee31ae0e63941405e3a8ac75a479d3cf" Jul 06 20:56:54.257267 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.257250 2582 scope.go:117] "RemoveContainer" containerID="9d983b28894a59129f6c4baecec5c514f8e729e0d666dab23f74363469909f8c" Jul 06 20:56:54.257503 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:56:54.257485 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d983b28894a59129f6c4baecec5c514f8e729e0d666dab23f74363469909f8c\": container with ID starting with 9d983b28894a59129f6c4baecec5c514f8e729e0d666dab23f74363469909f8c not found: ID does not exist" containerID="9d983b28894a59129f6c4baecec5c514f8e729e0d666dab23f74363469909f8c" Jul 06 20:56:54.257547 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.257513 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d983b28894a59129f6c4baecec5c514f8e729e0d666dab23f74363469909f8c"} err="failed to get container status \"9d983b28894a59129f6c4baecec5c514f8e729e0d666dab23f74363469909f8c\": rpc error: code = NotFound desc = could not find container \"9d983b28894a59129f6c4baecec5c514f8e729e0d666dab23f74363469909f8c\": container with ID starting with 9d983b28894a59129f6c4baecec5c514f8e729e0d666dab23f74363469909f8c not found: ID does not exist" Jul 06 20:56:54.257547 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.257532 2582 scope.go:117] "RemoveContainer" containerID="fdbdac8e2a5dfd934a389d00c54256f2ee31ae0e63941405e3a8ac75a479d3cf" Jul 06 20:56:54.257803 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:56:54.257785 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbdac8e2a5dfd934a389d00c54256f2ee31ae0e63941405e3a8ac75a479d3cf\": container with ID starting with fdbdac8e2a5dfd934a389d00c54256f2ee31ae0e63941405e3a8ac75a479d3cf not found: ID does not exist" containerID="fdbdac8e2a5dfd934a389d00c54256f2ee31ae0e63941405e3a8ac75a479d3cf" Jul 06 20:56:54.257853 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.257809 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbdac8e2a5dfd934a389d00c54256f2ee31ae0e63941405e3a8ac75a479d3cf"} err="failed to get container status \"fdbdac8e2a5dfd934a389d00c54256f2ee31ae0e63941405e3a8ac75a479d3cf\": rpc error: code = NotFound desc = could not find container \"fdbdac8e2a5dfd934a389d00c54256f2ee31ae0e63941405e3a8ac75a479d3cf\": container with ID starting with fdbdac8e2a5dfd934a389d00c54256f2ee31ae0e63941405e3a8ac75a479d3cf not found: ID does not exist" Jul 06 20:56:54.257853 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.257825 2582 scope.go:117] "RemoveContainer" containerID="df8a12397de10060af8fe855f43153cb1ef6d9562d60c94cf9ea6e136bfdb6e5" Jul 06 20:56:54.264973 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.264955 2582 scope.go:117] "RemoveContainer" containerID="88c9b2abcb11c8ce1ffdc192b2c23f4a344a4def752699543c0e2d73cb5b5b59" Jul 06 20:56:54.268111 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.268091 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2"] Jul 06 20:56:54.273108 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.273088 2582 scope.go:117] "RemoveContainer" containerID="df8a12397de10060af8fe855f43153cb1ef6d9562d60c94cf9ea6e136bfdb6e5" Jul 06 20:56:54.273179 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.273159 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87d4d-predictor-5dfdb8946b-9snv2"] Jul 06 20:56:54.273368 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:56:54.273350 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8a12397de10060af8fe855f43153cb1ef6d9562d60c94cf9ea6e136bfdb6e5\": container with ID starting with df8a12397de10060af8fe855f43153cb1ef6d9562d60c94cf9ea6e136bfdb6e5 not found: ID does not exist" containerID="df8a12397de10060af8fe855f43153cb1ef6d9562d60c94cf9ea6e136bfdb6e5" Jul 06 20:56:54.273409 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.273374 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8a12397de10060af8fe855f43153cb1ef6d9562d60c94cf9ea6e136bfdb6e5"} err="failed to get container status \"df8a12397de10060af8fe855f43153cb1ef6d9562d60c94cf9ea6e136bfdb6e5\": rpc error: code = NotFound desc = could not find container \"df8a12397de10060af8fe855f43153cb1ef6d9562d60c94cf9ea6e136bfdb6e5\": container with ID starting with df8a12397de10060af8fe855f43153cb1ef6d9562d60c94cf9ea6e136bfdb6e5 not found: ID does not exist" Jul 06 20:56:54.273409 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.273392 2582 scope.go:117] "RemoveContainer" containerID="88c9b2abcb11c8ce1ffdc192b2c23f4a344a4def752699543c0e2d73cb5b5b59" Jul 06 20:56:54.273630 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:56:54.273612 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c9b2abcb11c8ce1ffdc192b2c23f4a344a4def752699543c0e2d73cb5b5b59\": container with ID starting with 88c9b2abcb11c8ce1ffdc192b2c23f4a344a4def752699543c0e2d73cb5b5b59 not found: ID does not exist" containerID="88c9b2abcb11c8ce1ffdc192b2c23f4a344a4def752699543c0e2d73cb5b5b59" Jul 06 20:56:54.273684 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.273639 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c9b2abcb11c8ce1ffdc192b2c23f4a344a4def752699543c0e2d73cb5b5b59"} err="failed to get container status \"88c9b2abcb11c8ce1ffdc192b2c23f4a344a4def752699543c0e2d73cb5b5b59\": rpc error: code = NotFound desc = could not find container \"88c9b2abcb11c8ce1ffdc192b2c23f4a344a4def752699543c0e2d73cb5b5b59\": container with ID starting with 88c9b2abcb11c8ce1ffdc192b2c23f4a344a4def752699543c0e2d73cb5b5b59 not found: ID does not exist" Jul 06 20:56:54.286058 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.286031 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c"] Jul 06 20:56:54.293281 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:54.293252 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87d4d-predictor-779b4f5579-8cp6c"] Jul 06 20:56:55.538591 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:55.538555 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" path="/var/lib/kubelet/pods/40e7d7b2-3e68-46e5-ab5f-80aed349e5f3/volumes" Jul 06 20:56:55.539222 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:55.539200 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" path="/var/lib/kubelet/pods/ff6dd3d9-a337-45c3-a51a-218105f7ff8e/volumes" Jul 06 20:56:57.234822 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:57.234784 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:56:57.235385 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:57.235351 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 06 20:56:58.238545 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:58.238510 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:56:58.239167 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:56:58.239139 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" Jul 06 20:57:07.236139 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:07.236091 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 06 20:57:08.239816 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:08.239752 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" Jul 06 20:57:17.235344 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:17.235306 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 06 20:57:18.239327 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:18.239282 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" Jul 06 20:57:27.235961 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:27.235905 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 06 20:57:28.239680 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:28.239633 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" Jul 06 20:57:29.703331 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.703298 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq"] Jul 06 20:57:29.703882 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.703590 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kserve-container" containerID="cri-o://bd57a626e813bb9f91fda0b463007283d307eee53cdb79b4b561677dfe397e10" gracePeriod=30 Jul 06 20:57:29.703882 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.703617 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kube-rbac-proxy" containerID="cri-o://513a1f7d6890580eb71f71ec71255c22563c9c9efae81edec2e0fcbdd2bb1762" gracePeriod=30 Jul 06 20:57:29.829752 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.829716 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57"] Jul 06 20:57:29.830105 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.830092 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kube-rbac-proxy" Jul 06 20:57:29.830153 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.830107 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kube-rbac-proxy" Jul 06 20:57:29.830153 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.830116 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kube-rbac-proxy" Jul 06 20:57:29.830153 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.830121 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kube-rbac-proxy" Jul 06 20:57:29.830153 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.830131 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kserve-container" Jul 06 20:57:29.830153 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.830136 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kserve-container" Jul 06 20:57:29.830153 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.830142 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kserve-container" Jul 06 20:57:29.830153 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.830147 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kserve-container" Jul 06 20:57:29.830377 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.830199 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kube-rbac-proxy" Jul 06 20:57:29.830377 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.830208 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6dd3d9-a337-45c3-a51a-218105f7ff8e" containerName="kserve-container" Jul 06 20:57:29.830377 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.830215 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kserve-container" Jul 06 20:57:29.830377 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.830222 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e7d7b2-3e68-46e5-ab5f-80aed349e5f3" containerName="kube-rbac-proxy" Jul 06 20:57:29.833236 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.833214 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:29.844801 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.843986 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-24ddf-kube-rbac-proxy-sar-config\"" Jul 06 20:57:29.844801 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.844240 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-24ddf-predictor-serving-cert\"" Jul 06 20:57:29.869958 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.869923 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq"] Jul 06 20:57:29.870236 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.870196 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kserve-container" containerID="cri-o://89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b" gracePeriod=30 Jul 06 20:57:29.870322 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.870272 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kube-rbac-proxy" containerID="cri-o://5d5933fdf8107341c87866f9502aa83b85a6e0325d484c697569a8f9174a79b4" gracePeriod=30 Jul 06 20:57:29.984246 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:29.984159 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57"] Jul 06 20:57:30.025194 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.025160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc97a0d-30b2-4875-b3c6-931994dfca80-proxy-tls\") pod \"success-200-isvc-24ddf-predictor-b8f9d9964-25s57\" (UID: \"acc97a0d-30b2-4875-b3c6-931994dfca80\") " pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:30.025362 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.025221 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg6rx\" (UniqueName: \"kubernetes.io/projected/acc97a0d-30b2-4875-b3c6-931994dfca80-kube-api-access-mg6rx\") pod \"success-200-isvc-24ddf-predictor-b8f9d9964-25s57\" (UID: \"acc97a0d-30b2-4875-b3c6-931994dfca80\") " pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:30.025362 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.025254 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acc97a0d-30b2-4875-b3c6-931994dfca80-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-24ddf-predictor-b8f9d9964-25s57\" (UID: \"acc97a0d-30b2-4875-b3c6-931994dfca80\") " pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:30.078071 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.078037 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw"] Jul 06 20:57:30.081394 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.081376 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:30.084913 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.084884 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-24ddf-predictor-serving-cert\"" Jul 06 20:57:30.085308 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.085291 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-24ddf-kube-rbac-proxy-sar-config\"" Jul 06 20:57:30.098476 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.098442 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw"] Jul 06 20:57:30.126434 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.126406 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg6rx\" (UniqueName: \"kubernetes.io/projected/acc97a0d-30b2-4875-b3c6-931994dfca80-kube-api-access-mg6rx\") pod \"success-200-isvc-24ddf-predictor-b8f9d9964-25s57\" (UID: \"acc97a0d-30b2-4875-b3c6-931994dfca80\") " pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:30.126589 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.126451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acc97a0d-30b2-4875-b3c6-931994dfca80-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-24ddf-predictor-b8f9d9964-25s57\" (UID: \"acc97a0d-30b2-4875-b3c6-931994dfca80\") " pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:30.126589 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.126567 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc97a0d-30b2-4875-b3c6-931994dfca80-proxy-tls\") pod \"success-200-isvc-24ddf-predictor-b8f9d9964-25s57\" (UID: \"acc97a0d-30b2-4875-b3c6-931994dfca80\") " pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:30.127068 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.127050 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acc97a0d-30b2-4875-b3c6-931994dfca80-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-24ddf-predictor-b8f9d9964-25s57\" (UID: \"acc97a0d-30b2-4875-b3c6-931994dfca80\") " pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:30.128950 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.128922 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc97a0d-30b2-4875-b3c6-931994dfca80-proxy-tls\") pod \"success-200-isvc-24ddf-predictor-b8f9d9964-25s57\" (UID: \"acc97a0d-30b2-4875-b3c6-931994dfca80\") " pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:30.142545 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.142512 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg6rx\" (UniqueName: \"kubernetes.io/projected/acc97a0d-30b2-4875-b3c6-931994dfca80-kube-api-access-mg6rx\") pod \"success-200-isvc-24ddf-predictor-b8f9d9964-25s57\" (UID: \"acc97a0d-30b2-4875-b3c6-931994dfca80\") " pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:30.148205 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.148154 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:30.227687 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.227650 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjr5l\" (UniqueName: \"kubernetes.io/projected/58229cdc-1038-4689-96c7-428db5647f04-kube-api-access-pjr5l\") pod \"error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw\" (UID: \"58229cdc-1038-4689-96c7-428db5647f04\") " pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:30.228141 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.227749 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58229cdc-1038-4689-96c7-428db5647f04-proxy-tls\") pod \"error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw\" (UID: \"58229cdc-1038-4689-96c7-428db5647f04\") " pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:30.228141 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.227903 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58229cdc-1038-4689-96c7-428db5647f04-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw\" (UID: \"58229cdc-1038-4689-96c7-428db5647f04\") " pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:30.312682 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.312653 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57"] Jul 06 20:57:30.315261 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:57:30.315227 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc97a0d_30b2_4875_b3c6_931994dfca80.slice/crio-7961a4288248fc344e6c97383592e6b2b6610d1c8d7250aaa854a4398348f774 WatchSource:0}: Error finding container 7961a4288248fc344e6c97383592e6b2b6610d1c8d7250aaa854a4398348f774: Status 404 returned error can't find the container with id 7961a4288248fc344e6c97383592e6b2b6610d1c8d7250aaa854a4398348f774 Jul 06 20:57:30.329263 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.329228 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58229cdc-1038-4689-96c7-428db5647f04-proxy-tls\") pod \"error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw\" (UID: \"58229cdc-1038-4689-96c7-428db5647f04\") " pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:30.329367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.329278 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58229cdc-1038-4689-96c7-428db5647f04-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw\" (UID: \"58229cdc-1038-4689-96c7-428db5647f04\") " pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:30.329367 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.329309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjr5l\" (UniqueName: \"kubernetes.io/projected/58229cdc-1038-4689-96c7-428db5647f04-kube-api-access-pjr5l\") pod \"error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw\" (UID: \"58229cdc-1038-4689-96c7-428db5647f04\") " pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:30.329500 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:57:30.329369 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-serving-cert: secret "error-404-isvc-24ddf-predictor-serving-cert" not found Jul 06 20:57:30.329500 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:57:30.329420 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58229cdc-1038-4689-96c7-428db5647f04-proxy-tls podName:58229cdc-1038-4689-96c7-428db5647f04 nodeName:}" failed. No retries permitted until 2026-07-06 20:57:30.829405897 +0000 UTC m=+1317.914961068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/58229cdc-1038-4689-96c7-428db5647f04-proxy-tls") pod "error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" (UID: "58229cdc-1038-4689-96c7-428db5647f04") : secret "error-404-isvc-24ddf-predictor-serving-cert" not found Jul 06 20:57:30.329965 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.329946 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58229cdc-1038-4689-96c7-428db5647f04-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw\" (UID: \"58229cdc-1038-4689-96c7-428db5647f04\") " pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:30.346784 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.346745 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjr5l\" (UniqueName: \"kubernetes.io/projected/58229cdc-1038-4689-96c7-428db5647f04-kube-api-access-pjr5l\") pod \"error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw\" (UID: \"58229cdc-1038-4689-96c7-428db5647f04\") " pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:30.418510 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.418481 2582 generic.go:356] "Generic (PLEG): container finished" podID="18578650-14bd-4276-989b-41074feddf54" containerID="5d5933fdf8107341c87866f9502aa83b85a6e0325d484c697569a8f9174a79b4" exitCode=2 Jul 06 20:57:30.418623 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.418552 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" event={"ID":"18578650-14bd-4276-989b-41074feddf54","Type":"ContainerDied","Data":"5d5933fdf8107341c87866f9502aa83b85a6e0325d484c697569a8f9174a79b4"} Jul 06 20:57:30.419902 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.419877 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" event={"ID":"acc97a0d-30b2-4875-b3c6-931994dfca80","Type":"ContainerStarted","Data":"f87d5b76137dc32aec4b9974792ead58976b523153e38c5e51ce50c6d0b7bdcc"} Jul 06 20:57:30.420006 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.419942 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" event={"ID":"acc97a0d-30b2-4875-b3c6-931994dfca80","Type":"ContainerStarted","Data":"7961a4288248fc344e6c97383592e6b2b6610d1c8d7250aaa854a4398348f774"} Jul 06 20:57:30.421463 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.421445 2582 generic.go:356] "Generic (PLEG): container finished" podID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerID="513a1f7d6890580eb71f71ec71255c22563c9c9efae81edec2e0fcbdd2bb1762" exitCode=2 Jul 06 20:57:30.421559 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.421483 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" event={"ID":"ce90d61a-992b-4ab1-81e4-daeed0701f53","Type":"ContainerDied","Data":"513a1f7d6890580eb71f71ec71255c22563c9c9efae81edec2e0fcbdd2bb1762"} Jul 06 20:57:30.832409 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.832372 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58229cdc-1038-4689-96c7-428db5647f04-proxy-tls\") pod \"error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw\" (UID: \"58229cdc-1038-4689-96c7-428db5647f04\") " pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:30.834859 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.834830 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58229cdc-1038-4689-96c7-428db5647f04-proxy-tls\") pod \"error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw\" (UID: \"58229cdc-1038-4689-96c7-428db5647f04\") " pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:30.991116 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:30.991082 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:31.130643 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:31.130620 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw"] Jul 06 20:57:31.132635 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:57:31.132604 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58229cdc_1038_4689_96c7_428db5647f04.slice/crio-5aecd9597775d44ad0d6e7c98c1b49c3f5539428feb1a8d0bd98c14c5e779be3 WatchSource:0}: Error finding container 5aecd9597775d44ad0d6e7c98c1b49c3f5539428feb1a8d0bd98c14c5e779be3: Status 404 returned error can't find the container with id 5aecd9597775d44ad0d6e7c98c1b49c3f5539428feb1a8d0bd98c14c5e779be3 Jul 06 20:57:31.428086 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:31.427990 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" event={"ID":"58229cdc-1038-4689-96c7-428db5647f04","Type":"ContainerStarted","Data":"1b5ee6b8729d506ecce7ac85d831848ef7ce8f25aa010dd68bbf78b55e80201d"} Jul 06 20:57:31.428086 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:31.428052 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" event={"ID":"58229cdc-1038-4689-96c7-428db5647f04","Type":"ContainerStarted","Data":"eb81fb4dcf052ba79486cb99b0a86d8cbb8974039da65afed41d607cd99b9ca3"} Jul 06 20:57:31.428086 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:31.428073 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" event={"ID":"58229cdc-1038-4689-96c7-428db5647f04","Type":"ContainerStarted","Data":"5aecd9597775d44ad0d6e7c98c1b49c3f5539428feb1a8d0bd98c14c5e779be3"} Jul 06 20:57:31.428346 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:31.428198 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:31.429816 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:31.429790 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" event={"ID":"acc97a0d-30b2-4875-b3c6-931994dfca80","Type":"ContainerStarted","Data":"500e0a710912fc4cd47ca9510e54a45fa79200dd842f4461643e41123eeaa87b"} Jul 06 20:57:31.429936 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:31.429848 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:31.451772 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:31.451708 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" podStartSLOduration=1.451690776 podStartE2EDuration="1.451690776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:57:31.451141282 +0000 UTC m=+1318.536696477" watchObservedRunningTime="2026-07-06 20:57:31.451690776 +0000 UTC m=+1318.537245960" Jul 06 20:57:31.476637 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:31.476586 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" podStartSLOduration=2.47656836 podStartE2EDuration="2.47656836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:57:31.475242896 +0000 UTC m=+1318.560798090" watchObservedRunningTime="2026-07-06 20:57:31.47656836 +0000 UTC m=+1318.562123554" Jul 06 20:57:31.895819 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:31.895753 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused" Jul 06 20:57:31.901139 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:31.901116 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Jul 06 20:57:32.436936 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:32.436807 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:32.436936 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:32.436884 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:32.437666 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:32.437632 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" Jul 06 20:57:32.437848 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:32.437640 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused" Jul 06 20:57:33.426462 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.426431 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:57:33.441521 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.441476 2582 generic.go:356] "Generic (PLEG): container finished" podID="18578650-14bd-4276-989b-41074feddf54" containerID="89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b" exitCode=0 Jul 06 20:57:33.441695 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.441591 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" Jul 06 20:57:33.441695 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.441640 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" event={"ID":"18578650-14bd-4276-989b-41074feddf54","Type":"ContainerDied","Data":"89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b"} Jul 06 20:57:33.441839 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.441721 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq" event={"ID":"18578650-14bd-4276-989b-41074feddf54","Type":"ContainerDied","Data":"1aa0a7edad83e413d15a3b685be06f52d0181415c944b31dc31ce3e535bd59e7"} Jul 06 20:57:33.441839 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.441784 2582 scope.go:117] "RemoveContainer" containerID="5d5933fdf8107341c87866f9502aa83b85a6e0325d484c697569a8f9174a79b4" Jul 06 20:57:33.442501 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.442463 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" Jul 06 20:57:33.442735 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.442528 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused" Jul 06 20:57:33.451257 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.451223 2582 scope.go:117] "RemoveContainer" containerID="89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b" Jul 06 20:57:33.453632 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.453610 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t76q6\" (UniqueName: \"kubernetes.io/projected/18578650-14bd-4276-989b-41074feddf54-kube-api-access-t76q6\") pod \"18578650-14bd-4276-989b-41074feddf54\" (UID: \"18578650-14bd-4276-989b-41074feddf54\") " Jul 06 20:57:33.453732 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.453650 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18578650-14bd-4276-989b-41074feddf54-kube-rbac-proxy-sar-config\") pod \"18578650-14bd-4276-989b-41074feddf54\" (UID: \"18578650-14bd-4276-989b-41074feddf54\") " Jul 06 20:57:33.453732 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.453696 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18578650-14bd-4276-989b-41074feddf54-proxy-tls\") pod \"18578650-14bd-4276-989b-41074feddf54\" (UID: \"18578650-14bd-4276-989b-41074feddf54\") " Jul 06 20:57:33.454990 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.454958 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18578650-14bd-4276-989b-41074feddf54-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "18578650-14bd-4276-989b-41074feddf54" (UID: "18578650-14bd-4276-989b-41074feddf54"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:57:33.455911 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.455883 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18578650-14bd-4276-989b-41074feddf54-kube-api-access-t76q6" (OuterVolumeSpecName: "kube-api-access-t76q6") pod "18578650-14bd-4276-989b-41074feddf54" (UID: "18578650-14bd-4276-989b-41074feddf54"). InnerVolumeSpecName "kube-api-access-t76q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:57:33.456377 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.456358 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18578650-14bd-4276-989b-41074feddf54-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18578650-14bd-4276-989b-41074feddf54" (UID: "18578650-14bd-4276-989b-41074feddf54"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:57:33.468274 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.468252 2582 scope.go:117] "RemoveContainer" containerID="89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b" Jul 06 20:57:33.473431 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:57:33.473405 2582 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_kserve-container_success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq_kserve-ci-e2e-test_18578650-14bd-4276-989b-41074feddf54_0 in pod sandbox 1aa0a7edad83e413d15a3b685be06f52d0181415c944b31dc31ce3e535bd59e7 from index: no such id: '89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b'" containerID="89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b" Jul 06 20:57:33.473533 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.473444 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b"} err="rpc error: code = Unknown desc = failed to delete container k8s_kserve-container_success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq_kserve-ci-e2e-test_18578650-14bd-4276-989b-41074feddf54_0 in pod sandbox 1aa0a7edad83e413d15a3b685be06f52d0181415c944b31dc31ce3e535bd59e7 from index: no such id: '89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b'" Jul 06 20:57:33.473533 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.473469 2582 scope.go:117] "RemoveContainer" containerID="5d5933fdf8107341c87866f9502aa83b85a6e0325d484c697569a8f9174a79b4" Jul 06 20:57:33.473744 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:57:33.473728 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5933fdf8107341c87866f9502aa83b85a6e0325d484c697569a8f9174a79b4\": container with ID starting with 5d5933fdf8107341c87866f9502aa83b85a6e0325d484c697569a8f9174a79b4 not found: ID does not exist" containerID="5d5933fdf8107341c87866f9502aa83b85a6e0325d484c697569a8f9174a79b4" Jul 06 20:57:33.473871 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.473754 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5933fdf8107341c87866f9502aa83b85a6e0325d484c697569a8f9174a79b4"} err="failed to get container status \"5d5933fdf8107341c87866f9502aa83b85a6e0325d484c697569a8f9174a79b4\": rpc error: code = NotFound desc = could not find container \"5d5933fdf8107341c87866f9502aa83b85a6e0325d484c697569a8f9174a79b4\": container with ID starting with 5d5933fdf8107341c87866f9502aa83b85a6e0325d484c697569a8f9174a79b4 not found: ID does not exist" Jul 06 20:57:33.473871 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.473796 2582 scope.go:117] "RemoveContainer" containerID="89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b" Jul 06 20:57:33.474036 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:57:33.474021 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b\": container with ID starting with 89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b not found: ID does not exist" containerID="89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b" Jul 06 20:57:33.474089 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.474046 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b"} err="failed to get container status \"89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b\": rpc error: code = NotFound desc = could not find container \"89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b\": container with ID starting with 89d5b4d841088b1d15c596d1276ce2166f1fdb8714ff7d2acf374967a9083e7b not found: ID does not exist" Jul 06 20:57:33.554842 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.554817 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t76q6\" (UniqueName: \"kubernetes.io/projected/18578650-14bd-4276-989b-41074feddf54-kube-api-access-t76q6\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:57:33.554842 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.554839 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18578650-14bd-4276-989b-41074feddf54-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:57:33.554986 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.554851 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18578650-14bd-4276-989b-41074feddf54-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:57:33.773252 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.773224 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq"] Jul 06 20:57:33.775951 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.775929 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:57:33.780083 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.780056 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dfad7-predictor-dffd7f4cb-6ddvq"] Jul 06 20:57:33.861885 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.861855 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpqtm\" (UniqueName: \"kubernetes.io/projected/ce90d61a-992b-4ab1-81e4-daeed0701f53-kube-api-access-xpqtm\") pod \"ce90d61a-992b-4ab1-81e4-daeed0701f53\" (UID: \"ce90d61a-992b-4ab1-81e4-daeed0701f53\") " Jul 06 20:57:33.862043 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.861905 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce90d61a-992b-4ab1-81e4-daeed0701f53-proxy-tls\") pod \"ce90d61a-992b-4ab1-81e4-daeed0701f53\" (UID: \"ce90d61a-992b-4ab1-81e4-daeed0701f53\") " Jul 06 20:57:33.862043 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.861937 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce90d61a-992b-4ab1-81e4-daeed0701f53-kube-rbac-proxy-sar-config\") pod \"ce90d61a-992b-4ab1-81e4-daeed0701f53\" (UID: \"ce90d61a-992b-4ab1-81e4-daeed0701f53\") " Jul 06 20:57:33.862328 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.862306 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce90d61a-992b-4ab1-81e4-daeed0701f53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce90d61a-992b-4ab1-81e4-daeed0701f53" (UID: "ce90d61a-992b-4ab1-81e4-daeed0701f53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:57:33.864014 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.863986 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce90d61a-992b-4ab1-81e4-daeed0701f53-kube-api-access-xpqtm" (OuterVolumeSpecName: "kube-api-access-xpqtm") pod "ce90d61a-992b-4ab1-81e4-daeed0701f53" (UID: "ce90d61a-992b-4ab1-81e4-daeed0701f53"). InnerVolumeSpecName "kube-api-access-xpqtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:57:33.864014 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.864000 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce90d61a-992b-4ab1-81e4-daeed0701f53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce90d61a-992b-4ab1-81e4-daeed0701f53" (UID: "ce90d61a-992b-4ab1-81e4-daeed0701f53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:57:33.962729 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.962691 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpqtm\" (UniqueName: \"kubernetes.io/projected/ce90d61a-992b-4ab1-81e4-daeed0701f53-kube-api-access-xpqtm\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:57:33.962729 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.962722 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce90d61a-992b-4ab1-81e4-daeed0701f53-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:57:33.962729 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:33.962732 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce90d61a-992b-4ab1-81e4-daeed0701f53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:57:34.448643 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:34.448611 2582 generic.go:356] "Generic (PLEG): container finished" podID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerID="bd57a626e813bb9f91fda0b463007283d307eee53cdb79b4b561677dfe397e10" exitCode=0 Jul 06 20:57:34.449115 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:34.448659 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" Jul 06 20:57:34.449115 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:34.448688 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" event={"ID":"ce90d61a-992b-4ab1-81e4-daeed0701f53","Type":"ContainerDied","Data":"bd57a626e813bb9f91fda0b463007283d307eee53cdb79b4b561677dfe397e10"} Jul 06 20:57:34.449115 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:34.448743 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq" event={"ID":"ce90d61a-992b-4ab1-81e4-daeed0701f53","Type":"ContainerDied","Data":"67f28f515209abe606012447fb556ae6a782338083fe91ae94016167e3210e67"} Jul 06 20:57:34.449115 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:34.448799 2582 scope.go:117] "RemoveContainer" containerID="513a1f7d6890580eb71f71ec71255c22563c9c9efae81edec2e0fcbdd2bb1762" Jul 06 20:57:34.457110 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:34.457094 2582 scope.go:117] "RemoveContainer" containerID="bd57a626e813bb9f91fda0b463007283d307eee53cdb79b4b561677dfe397e10" Jul 06 20:57:34.464742 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:34.464725 2582 scope.go:117] "RemoveContainer" containerID="513a1f7d6890580eb71f71ec71255c22563c9c9efae81edec2e0fcbdd2bb1762" Jul 06 20:57:34.465021 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:57:34.465002 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"513a1f7d6890580eb71f71ec71255c22563c9c9efae81edec2e0fcbdd2bb1762\": container with ID starting with 513a1f7d6890580eb71f71ec71255c22563c9c9efae81edec2e0fcbdd2bb1762 not found: ID does not exist" containerID="513a1f7d6890580eb71f71ec71255c22563c9c9efae81edec2e0fcbdd2bb1762" Jul 06 20:57:34.465064 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:34.465030 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"513a1f7d6890580eb71f71ec71255c22563c9c9efae81edec2e0fcbdd2bb1762"} err="failed to get container status \"513a1f7d6890580eb71f71ec71255c22563c9c9efae81edec2e0fcbdd2bb1762\": rpc error: code = NotFound desc = could not find container \"513a1f7d6890580eb71f71ec71255c22563c9c9efae81edec2e0fcbdd2bb1762\": container with ID starting with 513a1f7d6890580eb71f71ec71255c22563c9c9efae81edec2e0fcbdd2bb1762 not found: ID does not exist" Jul 06 20:57:34.465064 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:34.465047 2582 scope.go:117] "RemoveContainer" containerID="bd57a626e813bb9f91fda0b463007283d307eee53cdb79b4b561677dfe397e10" Jul 06 20:57:34.465323 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:57:34.465304 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd57a626e813bb9f91fda0b463007283d307eee53cdb79b4b561677dfe397e10\": container with ID starting with bd57a626e813bb9f91fda0b463007283d307eee53cdb79b4b561677dfe397e10 not found: ID does not exist" containerID="bd57a626e813bb9f91fda0b463007283d307eee53cdb79b4b561677dfe397e10" Jul 06 20:57:34.465374 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:34.465328 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd57a626e813bb9f91fda0b463007283d307eee53cdb79b4b561677dfe397e10"} err="failed to get container status \"bd57a626e813bb9f91fda0b463007283d307eee53cdb79b4b561677dfe397e10\": rpc error: code = NotFound desc = could not find container \"bd57a626e813bb9f91fda0b463007283d307eee53cdb79b4b561677dfe397e10\": container with ID starting with bd57a626e813bb9f91fda0b463007283d307eee53cdb79b4b561677dfe397e10 not found: ID does not exist" Jul 06 20:57:34.478822 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:34.478787 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq"] Jul 06 20:57:34.481497 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:34.481475 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dfad7-predictor-9f88c844f-hddzq"] Jul 06 20:57:35.539869 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:35.539825 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18578650-14bd-4276-989b-41074feddf54" path="/var/lib/kubelet/pods/18578650-14bd-4276-989b-41074feddf54/volumes" Jul 06 20:57:35.540471 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:35.540446 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" path="/var/lib/kubelet/pods/ce90d61a-992b-4ab1-81e4-daeed0701f53/volumes" Jul 06 20:57:37.236301 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:37.236263 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" Jul 06 20:57:38.240844 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:38.240815 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:57:38.446636 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:38.446608 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:57:38.447141 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:38.447115 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused" Jul 06 20:57:38.447312 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:38.447296 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:57:38.447700 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:38.447677 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" Jul 06 20:57:47.235967 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:47.235937 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:57:48.447138 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:48.447102 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused" Jul 06 20:57:48.448331 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:48.448306 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" Jul 06 20:57:58.447420 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:58.447342 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused" Jul 06 20:57:58.447896 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:57:58.447643 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" Jul 06 20:58:08.447549 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:08.447511 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused" Jul 06 20:58:08.447974 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:08.447678 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" Jul 06 20:58:10.253579 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.253549 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf"] Jul 06 20:58:10.254000 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.253787 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kserve-container" containerID="cri-o://ed34c900b5b8c2b29aff2bbb00998672c38c40711e7d3749f27dd4aec1bd35cc" gracePeriod=30 Jul 06 20:58:10.254000 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.253892 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kube-rbac-proxy" containerID="cri-o://897b1a2f03f20cb6803bb5533189884cbdcf6aa2eea3c4af43a85d52e7f7d31b" gracePeriod=30 Jul 06 20:58:10.282667 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.282643 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt"] Jul 06 20:58:10.283069 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.283055 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kube-rbac-proxy" Jul 06 20:58:10.283116 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.283071 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kube-rbac-proxy" Jul 06 20:58:10.283116 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.283080 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kserve-container" Jul 06 20:58:10.283116 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.283085 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kserve-container" Jul 06 20:58:10.283116 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.283098 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kserve-container" Jul 06 20:58:10.283116 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.283104 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kserve-container" Jul 06 20:58:10.283116 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.283117 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kube-rbac-proxy" Jul 06 20:58:10.283325 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.283123 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kube-rbac-proxy" Jul 06 20:58:10.283325 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.283190 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kserve-container" Jul 06 20:58:10.283325 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.283199 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce90d61a-992b-4ab1-81e4-daeed0701f53" containerName="kube-rbac-proxy" Jul 06 20:58:10.283325 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.283206 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kserve-container" Jul 06 20:58:10.283325 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.283211 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="18578650-14bd-4276-989b-41074feddf54" containerName="kube-rbac-proxy" Jul 06 20:58:10.286452 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.286438 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:10.289336 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.289320 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-858c3-predictor-serving-cert\"" Jul 06 20:58:10.289411 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.289340 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-858c3-kube-rbac-proxy-sar-config\"" Jul 06 20:58:10.298066 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.298044 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt"] Jul 06 20:58:10.362724 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.362694 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0353baf0-e073-4f81-aa0c-68e01997550c-proxy-tls\") pod \"success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt\" (UID: \"0353baf0-e073-4f81-aa0c-68e01997550c\") " pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:10.362883 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.362814 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8ff2\" (UniqueName: \"kubernetes.io/projected/0353baf0-e073-4f81-aa0c-68e01997550c-kube-api-access-l8ff2\") pod \"success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt\" (UID: \"0353baf0-e073-4f81-aa0c-68e01997550c\") " pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:10.362883 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.362855 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0353baf0-e073-4f81-aa0c-68e01997550c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt\" (UID: \"0353baf0-e073-4f81-aa0c-68e01997550c\") " pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:10.399706 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.399676 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq"] Jul 06 20:58:10.403107 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.403089 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:10.407131 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.407106 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-858c3-kube-rbac-proxy-sar-config\"" Jul 06 20:58:10.407247 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.407124 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-858c3-predictor-serving-cert\"" Jul 06 20:58:10.415018 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.414991 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq"] Jul 06 20:58:10.443415 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.443380 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2"] Jul 06 20:58:10.443663 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.443637 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kserve-container" containerID="cri-o://841b4de1ae6e309fb10be1bc8eb97294546b0e653072b930ff53f8a78743e788" gracePeriod=30 Jul 06 20:58:10.444039 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.444015 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kube-rbac-proxy" containerID="cri-o://ab0cb4f30ce8c923fd1117eb82cb463cbe8743b852f418964a51f719a85de243" gracePeriod=30 Jul 06 20:58:10.464035 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.464006 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0353baf0-e073-4f81-aa0c-68e01997550c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt\" (UID: \"0353baf0-e073-4f81-aa0c-68e01997550c\") " pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:10.464142 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.464062 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0353baf0-e073-4f81-aa0c-68e01997550c-proxy-tls\") pod \"success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt\" (UID: \"0353baf0-e073-4f81-aa0c-68e01997550c\") " pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:10.464204 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.464145 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ff2\" (UniqueName: \"kubernetes.io/projected/0353baf0-e073-4f81-aa0c-68e01997550c-kube-api-access-l8ff2\") pod \"success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt\" (UID: \"0353baf0-e073-4f81-aa0c-68e01997550c\") " pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:10.464618 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.464601 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0353baf0-e073-4f81-aa0c-68e01997550c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt\" (UID: \"0353baf0-e073-4f81-aa0c-68e01997550c\") " pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:10.466312 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.466296 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0353baf0-e073-4f81-aa0c-68e01997550c-proxy-tls\") pod \"success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt\" (UID: \"0353baf0-e073-4f81-aa0c-68e01997550c\") " pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:10.476082 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.476058 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ff2\" (UniqueName: \"kubernetes.io/projected/0353baf0-e073-4f81-aa0c-68e01997550c-kube-api-access-l8ff2\") pod \"success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt\" (UID: \"0353baf0-e073-4f81-aa0c-68e01997550c\") " pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:10.565236 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.565208 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/290079df-dd51-4baa-8a85-08d49d7fcf25-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-858c3-predictor-6b7b999659-kmlhq\" (UID: \"290079df-dd51-4baa-8a85-08d49d7fcf25\") " pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:10.565350 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.565297 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/290079df-dd51-4baa-8a85-08d49d7fcf25-proxy-tls\") pod \"error-404-isvc-858c3-predictor-6b7b999659-kmlhq\" (UID: \"290079df-dd51-4baa-8a85-08d49d7fcf25\") " pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:10.565350 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.565332 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsg8p\" (UniqueName: \"kubernetes.io/projected/290079df-dd51-4baa-8a85-08d49d7fcf25-kube-api-access-gsg8p\") pod \"error-404-isvc-858c3-predictor-6b7b999659-kmlhq\" (UID: \"290079df-dd51-4baa-8a85-08d49d7fcf25\") " pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:10.596432 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.596396 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:10.621447 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.621418 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerID="897b1a2f03f20cb6803bb5533189884cbdcf6aa2eea3c4af43a85d52e7f7d31b" exitCode=2 Jul 06 20:58:10.621607 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.621490 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" event={"ID":"d4b9da3e-0e8f-4fd4-9550-82b899e84d89","Type":"ContainerDied","Data":"897b1a2f03f20cb6803bb5533189884cbdcf6aa2eea3c4af43a85d52e7f7d31b"} Jul 06 20:58:10.624528 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.624483 2582 generic.go:356] "Generic (PLEG): container finished" podID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerID="ab0cb4f30ce8c923fd1117eb82cb463cbe8743b852f418964a51f719a85de243" exitCode=2 Jul 06 20:58:10.624661 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.624588 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" event={"ID":"5d6f37bd-f266-4ce2-b814-c84d82b52b2a","Type":"ContainerDied","Data":"ab0cb4f30ce8c923fd1117eb82cb463cbe8743b852f418964a51f719a85de243"} Jul 06 20:58:10.666046 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.665998 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/290079df-dd51-4baa-8a85-08d49d7fcf25-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-858c3-predictor-6b7b999659-kmlhq\" (UID: \"290079df-dd51-4baa-8a85-08d49d7fcf25\") " pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:10.666164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.666098 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/290079df-dd51-4baa-8a85-08d49d7fcf25-proxy-tls\") pod \"error-404-isvc-858c3-predictor-6b7b999659-kmlhq\" (UID: \"290079df-dd51-4baa-8a85-08d49d7fcf25\") " pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:10.666164 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.666121 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsg8p\" (UniqueName: \"kubernetes.io/projected/290079df-dd51-4baa-8a85-08d49d7fcf25-kube-api-access-gsg8p\") pod \"error-404-isvc-858c3-predictor-6b7b999659-kmlhq\" (UID: \"290079df-dd51-4baa-8a85-08d49d7fcf25\") " pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:10.666384 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:58:10.666365 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-858c3-predictor-serving-cert: secret "error-404-isvc-858c3-predictor-serving-cert" not found Jul 06 20:58:10.666521 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:58:10.666509 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/290079df-dd51-4baa-8a85-08d49d7fcf25-proxy-tls podName:290079df-dd51-4baa-8a85-08d49d7fcf25 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:11.166485417 +0000 UTC m=+1358.252040590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/290079df-dd51-4baa-8a85-08d49d7fcf25-proxy-tls") pod "error-404-isvc-858c3-predictor-6b7b999659-kmlhq" (UID: "290079df-dd51-4baa-8a85-08d49d7fcf25") : secret "error-404-isvc-858c3-predictor-serving-cert" not found Jul 06 20:58:10.667103 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.667076 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/290079df-dd51-4baa-8a85-08d49d7fcf25-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-858c3-predictor-6b7b999659-kmlhq\" (UID: \"290079df-dd51-4baa-8a85-08d49d7fcf25\") " pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:10.680160 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.680140 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsg8p\" (UniqueName: \"kubernetes.io/projected/290079df-dd51-4baa-8a85-08d49d7fcf25-kube-api-access-gsg8p\") pod \"error-404-isvc-858c3-predictor-6b7b999659-kmlhq\" (UID: \"290079df-dd51-4baa-8a85-08d49d7fcf25\") " pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:10.737113 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:10.737074 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt"] Jul 06 20:58:10.741432 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:58:10.740472 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0353baf0_e073_4f81_aa0c_68e01997550c.slice/crio-bc81b8c3ad12dc1fbcd44867dc543cac76e0e49f3adae8e57531be2cc4445112 WatchSource:0}: Error finding container bc81b8c3ad12dc1fbcd44867dc543cac76e0e49f3adae8e57531be2cc4445112: Status 404 returned error can't find the container with id bc81b8c3ad12dc1fbcd44867dc543cac76e0e49f3adae8e57531be2cc4445112 Jul 06 20:58:11.170795 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.170742 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/290079df-dd51-4baa-8a85-08d49d7fcf25-proxy-tls\") pod \"error-404-isvc-858c3-predictor-6b7b999659-kmlhq\" (UID: \"290079df-dd51-4baa-8a85-08d49d7fcf25\") " pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:11.173270 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.173242 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/290079df-dd51-4baa-8a85-08d49d7fcf25-proxy-tls\") pod \"error-404-isvc-858c3-predictor-6b7b999659-kmlhq\" (UID: \"290079df-dd51-4baa-8a85-08d49d7fcf25\") " pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:11.316439 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.316405 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:11.456272 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.456100 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq"] Jul 06 20:58:11.459063 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:58:11.459031 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290079df_dd51_4baa_8a85_08d49d7fcf25.slice/crio-2c522ad44f7a6d77584423698b3e26e3bc75185e09939a276f3fbebef4134d42 WatchSource:0}: Error finding container 2c522ad44f7a6d77584423698b3e26e3bc75185e09939a276f3fbebef4134d42: Status 404 returned error can't find the container with id 2c522ad44f7a6d77584423698b3e26e3bc75185e09939a276f3fbebef4134d42 Jul 06 20:58:11.630958 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.630921 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" event={"ID":"0353baf0-e073-4f81-aa0c-68e01997550c","Type":"ContainerStarted","Data":"a0bc502a26f272992c15d7e6e95cb4fc91e85484ba72363067b05e24fbe7a3ed"} Jul 06 20:58:11.631119 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.630993 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:11.631201 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.631145 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:11.631201 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.631195 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" event={"ID":"0353baf0-e073-4f81-aa0c-68e01997550c","Type":"ContainerStarted","Data":"6d8e6e7f9b25fe0e41c4ed7a922c6f7e486454f638ab7f3231e79a7546eabd9c"} Jul 06 20:58:11.631314 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.631236 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" event={"ID":"0353baf0-e073-4f81-aa0c-68e01997550c","Type":"ContainerStarted","Data":"bc81b8c3ad12dc1fbcd44867dc543cac76e0e49f3adae8e57531be2cc4445112"} Jul 06 20:58:11.632813 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.632785 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused" Jul 06 20:58:11.632932 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.632840 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" event={"ID":"290079df-dd51-4baa-8a85-08d49d7fcf25","Type":"ContainerStarted","Data":"8e172d172a02a635c50fe9d40139cb33b8f01019648088f3ddd74b0ecf4c16e7"} Jul 06 20:58:11.632932 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.632887 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" event={"ID":"290079df-dd51-4baa-8a85-08d49d7fcf25","Type":"ContainerStarted","Data":"45f5835189f3369d4e17a6afea56461d74ec2684aeeb2d776261a7a6669bdeb5"} Jul 06 20:58:11.632932 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.632924 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" event={"ID":"290079df-dd51-4baa-8a85-08d49d7fcf25","Type":"ContainerStarted","Data":"2c522ad44f7a6d77584423698b3e26e3bc75185e09939a276f3fbebef4134d42"} Jul 06 20:58:11.633067 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.632959 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:11.653608 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.653549 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" podStartSLOduration=1.653533344 podStartE2EDuration="1.653533344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:58:11.651439668 +0000 UTC m=+1358.736994865" watchObservedRunningTime="2026-07-06 20:58:11.653533344 +0000 UTC m=+1358.739088537" Jul 06 20:58:11.673829 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:11.673770 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" podStartSLOduration=1.6737385599999999 podStartE2EDuration="1.67373856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:58:11.671373782 +0000 UTC m=+1358.756928976" watchObservedRunningTime="2026-07-06 20:58:11.67373856 +0000 UTC m=+1358.759293754" Jul 06 20:58:12.230888 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:12.230845 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused" Jul 06 20:58:12.640861 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:12.640830 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:12.641496 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:12.641406 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused" Jul 06 20:58:12.641929 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:12.641905 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Jul 06 20:58:13.235031 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:13.234977 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused" Jul 06 20:58:13.646508 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:13.646473 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Jul 06 20:58:14.209147 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.209121 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:58:14.296977 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.296894 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-kube-rbac-proxy-sar-config\") pod \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\" (UID: \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\") " Jul 06 20:58:14.296977 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.296943 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-proxy-tls\") pod \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\" (UID: \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\") " Jul 06 20:58:14.297140 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.296993 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs2ps\" (UniqueName: \"kubernetes.io/projected/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-kube-api-access-cs2ps\") pod \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\" (UID: \"d4b9da3e-0e8f-4fd4-9550-82b899e84d89\") " Jul 06 20:58:14.297296 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.297268 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4b9da3e-0e8f-4fd4-9550-82b899e84d89" (UID: "d4b9da3e-0e8f-4fd4-9550-82b899e84d89"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:58:14.298950 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.298921 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-kube-api-access-cs2ps" (OuterVolumeSpecName: "kube-api-access-cs2ps") pod "d4b9da3e-0e8f-4fd4-9550-82b899e84d89" (UID: "d4b9da3e-0e8f-4fd4-9550-82b899e84d89"). InnerVolumeSpecName "kube-api-access-cs2ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:58:14.299064 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.298954 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4b9da3e-0e8f-4fd4-9550-82b899e84d89" (UID: "d4b9da3e-0e8f-4fd4-9550-82b899e84d89"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:58:14.398303 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.398276 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:58:14.398303 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.398302 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:58:14.398475 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.398316 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs2ps\" (UniqueName: \"kubernetes.io/projected/d4b9da3e-0e8f-4fd4-9550-82b899e84d89-kube-api-access-cs2ps\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:58:14.489028 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.489003 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:58:14.599869 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.599846 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv6hc\" (UniqueName: \"kubernetes.io/projected/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-kube-api-access-nv6hc\") pod \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\" (UID: \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\") " Jul 06 20:58:14.600005 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.599904 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-kube-rbac-proxy-sar-config\") pod \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\" (UID: \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\") " Jul 06 20:58:14.600005 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.599934 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-proxy-tls\") pod \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\" (UID: \"5d6f37bd-f266-4ce2-b814-c84d82b52b2a\") " Jul 06 20:58:14.600273 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.600243 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5d6f37bd-f266-4ce2-b814-c84d82b52b2a" (UID: "5d6f37bd-f266-4ce2-b814-c84d82b52b2a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:58:14.601930 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.601902 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5d6f37bd-f266-4ce2-b814-c84d82b52b2a" (UID: "5d6f37bd-f266-4ce2-b814-c84d82b52b2a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:58:14.601930 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.601914 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-kube-api-access-nv6hc" (OuterVolumeSpecName: "kube-api-access-nv6hc") pod "5d6f37bd-f266-4ce2-b814-c84d82b52b2a" (UID: "5d6f37bd-f266-4ce2-b814-c84d82b52b2a"). InnerVolumeSpecName "kube-api-access-nv6hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:58:14.653516 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.653479 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerID="ed34c900b5b8c2b29aff2bbb00998672c38c40711e7d3749f27dd4aec1bd35cc" exitCode=0 Jul 06 20:58:14.653970 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.653522 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" Jul 06 20:58:14.653970 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.653549 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" event={"ID":"d4b9da3e-0e8f-4fd4-9550-82b899e84d89","Type":"ContainerDied","Data":"ed34c900b5b8c2b29aff2bbb00998672c38c40711e7d3749f27dd4aec1bd35cc"} Jul 06 20:58:14.653970 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.653608 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf" event={"ID":"d4b9da3e-0e8f-4fd4-9550-82b899e84d89","Type":"ContainerDied","Data":"b8dd057c02b5ecb5a16186f2c366deae98a66889676d97703b4d7bee01ff80eb"} Jul 06 20:58:14.653970 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.653634 2582 scope.go:117] "RemoveContainer" containerID="897b1a2f03f20cb6803bb5533189884cbdcf6aa2eea3c4af43a85d52e7f7d31b" Jul 06 20:58:14.655523 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.655497 2582 generic.go:356] "Generic (PLEG): container finished" podID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerID="841b4de1ae6e309fb10be1bc8eb97294546b0e653072b930ff53f8a78743e788" exitCode=0 Jul 06 20:58:14.655613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.655548 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" event={"ID":"5d6f37bd-f266-4ce2-b814-c84d82b52b2a","Type":"ContainerDied","Data":"841b4de1ae6e309fb10be1bc8eb97294546b0e653072b930ff53f8a78743e788"} Jul 06 20:58:14.655613 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.655599 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" event={"ID":"5d6f37bd-f266-4ce2-b814-c84d82b52b2a","Type":"ContainerDied","Data":"301a46dcd6b0fa5ae92e1bf98c540fbd81ceaa8d0f36934f1874d61e73abc6ec"} Jul 06 20:58:14.655700 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.655552 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2" Jul 06 20:58:14.662773 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.662741 2582 scope.go:117] "RemoveContainer" containerID="ed34c900b5b8c2b29aff2bbb00998672c38c40711e7d3749f27dd4aec1bd35cc" Jul 06 20:58:14.671123 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.671106 2582 scope.go:117] "RemoveContainer" containerID="897b1a2f03f20cb6803bb5533189884cbdcf6aa2eea3c4af43a85d52e7f7d31b" Jul 06 20:58:14.671399 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:58:14.671381 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897b1a2f03f20cb6803bb5533189884cbdcf6aa2eea3c4af43a85d52e7f7d31b\": container with ID starting with 897b1a2f03f20cb6803bb5533189884cbdcf6aa2eea3c4af43a85d52e7f7d31b not found: ID does not exist" containerID="897b1a2f03f20cb6803bb5533189884cbdcf6aa2eea3c4af43a85d52e7f7d31b" Jul 06 20:58:14.671476 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.671411 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897b1a2f03f20cb6803bb5533189884cbdcf6aa2eea3c4af43a85d52e7f7d31b"} err="failed to get container status \"897b1a2f03f20cb6803bb5533189884cbdcf6aa2eea3c4af43a85d52e7f7d31b\": rpc error: code = NotFound desc = could not find container \"897b1a2f03f20cb6803bb5533189884cbdcf6aa2eea3c4af43a85d52e7f7d31b\": container with ID starting with 897b1a2f03f20cb6803bb5533189884cbdcf6aa2eea3c4af43a85d52e7f7d31b not found: ID does not exist" Jul 06 20:58:14.671476 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.671435 2582 scope.go:117] "RemoveContainer" containerID="ed34c900b5b8c2b29aff2bbb00998672c38c40711e7d3749f27dd4aec1bd35cc" Jul 06 20:58:14.671690 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:58:14.671675 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed34c900b5b8c2b29aff2bbb00998672c38c40711e7d3749f27dd4aec1bd35cc\": container with ID starting with ed34c900b5b8c2b29aff2bbb00998672c38c40711e7d3749f27dd4aec1bd35cc not found: ID does not exist" containerID="ed34c900b5b8c2b29aff2bbb00998672c38c40711e7d3749f27dd4aec1bd35cc" Jul 06 20:58:14.671752 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.671701 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed34c900b5b8c2b29aff2bbb00998672c38c40711e7d3749f27dd4aec1bd35cc"} err="failed to get container status \"ed34c900b5b8c2b29aff2bbb00998672c38c40711e7d3749f27dd4aec1bd35cc\": rpc error: code = NotFound desc = could not find container \"ed34c900b5b8c2b29aff2bbb00998672c38c40711e7d3749f27dd4aec1bd35cc\": container with ID starting with ed34c900b5b8c2b29aff2bbb00998672c38c40711e7d3749f27dd4aec1bd35cc not found: ID does not exist" Jul 06 20:58:14.671752 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.671722 2582 scope.go:117] "RemoveContainer" containerID="ab0cb4f30ce8c923fd1117eb82cb463cbe8743b852f418964a51f719a85de243" Jul 06 20:58:14.680368 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.680345 2582 scope.go:117] "RemoveContainer" containerID="841b4de1ae6e309fb10be1bc8eb97294546b0e653072b930ff53f8a78743e788" Jul 06 20:58:14.680429 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.680390 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf"] Jul 06 20:58:14.684045 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.684025 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-305f4-predictor-c7fb94cc6-66jqf"] Jul 06 20:58:14.688162 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.688145 2582 scope.go:117] "RemoveContainer" containerID="ab0cb4f30ce8c923fd1117eb82cb463cbe8743b852f418964a51f719a85de243" Jul 06 20:58:14.688394 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:58:14.688375 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0cb4f30ce8c923fd1117eb82cb463cbe8743b852f418964a51f719a85de243\": container with ID starting with ab0cb4f30ce8c923fd1117eb82cb463cbe8743b852f418964a51f719a85de243 not found: ID does not exist" containerID="ab0cb4f30ce8c923fd1117eb82cb463cbe8743b852f418964a51f719a85de243" Jul 06 20:58:14.688453 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.688400 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0cb4f30ce8c923fd1117eb82cb463cbe8743b852f418964a51f719a85de243"} err="failed to get container status \"ab0cb4f30ce8c923fd1117eb82cb463cbe8743b852f418964a51f719a85de243\": rpc error: code = NotFound desc = could not find container \"ab0cb4f30ce8c923fd1117eb82cb463cbe8743b852f418964a51f719a85de243\": container with ID starting with ab0cb4f30ce8c923fd1117eb82cb463cbe8743b852f418964a51f719a85de243 not found: ID does not exist" Jul 06 20:58:14.688453 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.688416 2582 scope.go:117] "RemoveContainer" containerID="841b4de1ae6e309fb10be1bc8eb97294546b0e653072b930ff53f8a78743e788" Jul 06 20:58:14.688681 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:58:14.688666 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841b4de1ae6e309fb10be1bc8eb97294546b0e653072b930ff53f8a78743e788\": container with ID starting with 841b4de1ae6e309fb10be1bc8eb97294546b0e653072b930ff53f8a78743e788 not found: ID does not exist" containerID="841b4de1ae6e309fb10be1bc8eb97294546b0e653072b930ff53f8a78743e788" Jul 06 20:58:14.688732 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.688683 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841b4de1ae6e309fb10be1bc8eb97294546b0e653072b930ff53f8a78743e788"} err="failed to get container status \"841b4de1ae6e309fb10be1bc8eb97294546b0e653072b930ff53f8a78743e788\": rpc error: code = NotFound desc = could not find container \"841b4de1ae6e309fb10be1bc8eb97294546b0e653072b930ff53f8a78743e788\": container with ID starting with 841b4de1ae6e309fb10be1bc8eb97294546b0e653072b930ff53f8a78743e788 not found: ID does not exist" Jul 06 20:58:14.697690 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.697668 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2"] Jul 06 20:58:14.700702 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.700677 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv6hc\" (UniqueName: \"kubernetes.io/projected/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-kube-api-access-nv6hc\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:58:14.700815 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.700704 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:58:14.700815 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.700719 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d6f37bd-f266-4ce2-b814-c84d82b52b2a-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:58:14.702296 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:14.702277 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-305f4-predictor-5d5987d4f-pz2z2"] Jul 06 20:58:15.539423 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:15.539380 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" path="/var/lib/kubelet/pods/5d6f37bd-f266-4ce2-b814-c84d82b52b2a/volumes" Jul 06 20:58:15.540032 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:15.540008 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" path="/var/lib/kubelet/pods/d4b9da3e-0e8f-4fd4-9550-82b899e84d89/volumes" Jul 06 20:58:17.644773 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:17.644730 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:17.645233 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:17.645209 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused" Jul 06 20:58:18.447918 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:18.447890 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:58:18.448844 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:18.448824 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:58:18.651133 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:18.651102 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:18.651485 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:18.651462 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Jul 06 20:58:27.645398 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:27.645351 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused" Jul 06 20:58:28.651736 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:28.651690 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Jul 06 20:58:37.645686 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:37.645646 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused" Jul 06 20:58:38.652267 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:38.652221 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Jul 06 20:58:40.044055 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.044026 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt"] Jul 06 20:58:40.044458 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.044442 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kserve-container" Jul 06 20:58:40.044523 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.044462 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kserve-container" Jul 06 20:58:40.044523 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.044487 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kube-rbac-proxy" Jul 06 20:58:40.044523 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.044495 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kube-rbac-proxy" Jul 06 20:58:40.044523 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.044515 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kserve-container" Jul 06 20:58:40.044657 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.044524 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kserve-container" Jul 06 20:58:40.044657 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.044535 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kube-rbac-proxy" Jul 06 20:58:40.044657 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.044542 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kube-rbac-proxy" Jul 06 20:58:40.044657 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.044621 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kserve-container" Jul 06 20:58:40.044657 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.044634 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kserve-container" Jul 06 20:58:40.044657 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.044642 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b9da3e-0e8f-4fd4-9550-82b899e84d89" containerName="kube-rbac-proxy" Jul 06 20:58:40.044657 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.044652 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d6f37bd-f266-4ce2-b814-c84d82b52b2a" containerName="kube-rbac-proxy" Jul 06 20:58:40.049034 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.049017 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:40.053260 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.053239 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-92a61-predictor-serving-cert\"" Jul 06 20:58:40.053260 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.053257 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-92a61-kube-rbac-proxy-sar-config\"" Jul 06 20:58:40.069960 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.069940 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt"] Jul 06 20:58:40.107508 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.107485 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdcfb\" (UniqueName: \"kubernetes.io/projected/6611d492-131b-4cb5-81e2-e67702aac1f2-kube-api-access-gdcfb\") pod \"success-200-isvc-92a61-predictor-f78d96bf8-l44xt\" (UID: \"6611d492-131b-4cb5-81e2-e67702aac1f2\") " pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:40.107639 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.107517 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6611d492-131b-4cb5-81e2-e67702aac1f2-proxy-tls\") pod \"success-200-isvc-92a61-predictor-f78d96bf8-l44xt\" (UID: \"6611d492-131b-4cb5-81e2-e67702aac1f2\") " pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:40.107639 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.107616 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6611d492-131b-4cb5-81e2-e67702aac1f2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-92a61-predictor-f78d96bf8-l44xt\" (UID: \"6611d492-131b-4cb5-81e2-e67702aac1f2\") " pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:40.128298 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.128260 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57"] Jul 06 20:58:40.128513 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.128490 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kube-rbac-proxy" containerID="cri-o://500e0a710912fc4cd47ca9510e54a45fa79200dd842f4461643e41123eeaa87b" gracePeriod=30 Jul 06 20:58:40.128598 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.128471 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kserve-container" containerID="cri-o://f87d5b76137dc32aec4b9974792ead58976b523153e38c5e51ce50c6d0b7bdcc" gracePeriod=30 Jul 06 20:58:40.208693 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.208663 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdcfb\" (UniqueName: \"kubernetes.io/projected/6611d492-131b-4cb5-81e2-e67702aac1f2-kube-api-access-gdcfb\") pod \"success-200-isvc-92a61-predictor-f78d96bf8-l44xt\" (UID: \"6611d492-131b-4cb5-81e2-e67702aac1f2\") " pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:40.208840 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.208701 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6611d492-131b-4cb5-81e2-e67702aac1f2-proxy-tls\") pod \"success-200-isvc-92a61-predictor-f78d96bf8-l44xt\" (UID: \"6611d492-131b-4cb5-81e2-e67702aac1f2\") " pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:40.208840 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.208755 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6611d492-131b-4cb5-81e2-e67702aac1f2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-92a61-predictor-f78d96bf8-l44xt\" (UID: \"6611d492-131b-4cb5-81e2-e67702aac1f2\") " pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:40.209482 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.209464 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6611d492-131b-4cb5-81e2-e67702aac1f2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-92a61-predictor-f78d96bf8-l44xt\" (UID: \"6611d492-131b-4cb5-81e2-e67702aac1f2\") " pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:40.211216 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.211192 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6611d492-131b-4cb5-81e2-e67702aac1f2-proxy-tls\") pod \"success-200-isvc-92a61-predictor-f78d96bf8-l44xt\" (UID: \"6611d492-131b-4cb5-81e2-e67702aac1f2\") " pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:40.212696 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.212679 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv"] Jul 06 20:58:40.216223 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.216207 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:40.219813 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.219792 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-92a61-predictor-serving-cert\"" Jul 06 20:58:40.221342 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.221320 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdcfb\" (UniqueName: \"kubernetes.io/projected/6611d492-131b-4cb5-81e2-e67702aac1f2-kube-api-access-gdcfb\") pod \"success-200-isvc-92a61-predictor-f78d96bf8-l44xt\" (UID: \"6611d492-131b-4cb5-81e2-e67702aac1f2\") " pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:40.221449 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.221436 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-92a61-kube-rbac-proxy-sar-config\"" Jul 06 20:58:40.237494 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.237469 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv"] Jul 06 20:58:40.298448 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.298375 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw"] Jul 06 20:58:40.298628 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.298587 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kserve-container" containerID="cri-o://eb81fb4dcf052ba79486cb99b0a86d8cbb8974039da65afed41d607cd99b9ca3" gracePeriod=30 Jul 06 20:58:40.298700 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.298618 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kube-rbac-proxy" containerID="cri-o://1b5ee6b8729d506ecce7ac85d831848ef7ce8f25aa010dd68bbf78b55e80201d" gracePeriod=30 Jul 06 20:58:40.310099 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.310076 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc66437a-9311-48cd-963f-ac035c17b4a3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv\" (UID: \"fc66437a-9311-48cd-963f-ac035c17b4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:40.310176 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.310124 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfl92\" (UniqueName: \"kubernetes.io/projected/fc66437a-9311-48cd-963f-ac035c17b4a3-kube-api-access-xfl92\") pod \"error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv\" (UID: \"fc66437a-9311-48cd-963f-ac035c17b4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:40.310222 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.310190 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc66437a-9311-48cd-963f-ac035c17b4a3-proxy-tls\") pod \"error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv\" (UID: \"fc66437a-9311-48cd-963f-ac035c17b4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:40.358346 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.358318 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:40.411706 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.411671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc66437a-9311-48cd-963f-ac035c17b4a3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv\" (UID: \"fc66437a-9311-48cd-963f-ac035c17b4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:40.411860 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.411735 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfl92\" (UniqueName: \"kubernetes.io/projected/fc66437a-9311-48cd-963f-ac035c17b4a3-kube-api-access-xfl92\") pod \"error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv\" (UID: \"fc66437a-9311-48cd-963f-ac035c17b4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:40.411860 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.411805 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc66437a-9311-48cd-963f-ac035c17b4a3-proxy-tls\") pod \"error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv\" (UID: \"fc66437a-9311-48cd-963f-ac035c17b4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:40.412291 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.412270 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc66437a-9311-48cd-963f-ac035c17b4a3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv\" (UID: \"fc66437a-9311-48cd-963f-ac035c17b4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:40.414386 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.414359 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc66437a-9311-48cd-963f-ac035c17b4a3-proxy-tls\") pod \"error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv\" (UID: \"fc66437a-9311-48cd-963f-ac035c17b4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:40.429585 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.429557 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfl92\" (UniqueName: \"kubernetes.io/projected/fc66437a-9311-48cd-963f-ac035c17b4a3-kube-api-access-xfl92\") pod \"error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv\" (UID: \"fc66437a-9311-48cd-963f-ac035c17b4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:40.505065 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.505037 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt"] Jul 06 20:58:40.505195 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:58:40.505168 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6611d492_131b_4cb5_81e2_e67702aac1f2.slice/crio-ce585c6dfc0ca706639074356c0a9eb5ca8ca7b625d8a9538621ff175b712800 WatchSource:0}: Error finding container ce585c6dfc0ca706639074356c0a9eb5ca8ca7b625d8a9538621ff175b712800: Status 404 returned error can't find the container with id ce585c6dfc0ca706639074356c0a9eb5ca8ca7b625d8a9538621ff175b712800 Jul 06 20:58:40.534249 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.534228 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:40.685012 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.684982 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv"] Jul 06 20:58:40.688560 ip-10-0-142-142 kubenswrapper[2582]: W0706 20:58:40.688528 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc66437a_9311_48cd_963f_ac035c17b4a3.slice/crio-15a062168b3296d3c38daa68dfd4dc4325a79e37b76445da351676a94e816a39 WatchSource:0}: Error finding container 15a062168b3296d3c38daa68dfd4dc4325a79e37b76445da351676a94e816a39: Status 404 returned error can't find the container with id 15a062168b3296d3c38daa68dfd4dc4325a79e37b76445da351676a94e816a39 Jul 06 20:58:40.776107 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.776080 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" event={"ID":"fc66437a-9311-48cd-963f-ac035c17b4a3","Type":"ContainerStarted","Data":"6d87cf3a52332597f96a1bd6e76ca5a4b8b1854f7f0b4f3afde9762a5dd8db9a"} Jul 06 20:58:40.776198 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.776143 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" event={"ID":"fc66437a-9311-48cd-963f-ac035c17b4a3","Type":"ContainerStarted","Data":"15a062168b3296d3c38daa68dfd4dc4325a79e37b76445da351676a94e816a39"} Jul 06 20:58:40.777808 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.777785 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" event={"ID":"6611d492-131b-4cb5-81e2-e67702aac1f2","Type":"ContainerStarted","Data":"6cc3b399501e28eeada8f6c90eb1c0972cd9efd7e607530dbf78d133f7732bcd"} Jul 06 20:58:40.777907 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.777839 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" event={"ID":"6611d492-131b-4cb5-81e2-e67702aac1f2","Type":"ContainerStarted","Data":"3d361b7695f739d4ac42643c2ed06aab46519d1e90756b4b494c6c6603ebb4ef"} Jul 06 20:58:40.777907 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.777867 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" event={"ID":"6611d492-131b-4cb5-81e2-e67702aac1f2","Type":"ContainerStarted","Data":"ce585c6dfc0ca706639074356c0a9eb5ca8ca7b625d8a9538621ff175b712800"} Jul 06 20:58:40.777907 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.777901 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:40.779801 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.779778 2582 generic.go:356] "Generic (PLEG): container finished" podID="58229cdc-1038-4689-96c7-428db5647f04" containerID="1b5ee6b8729d506ecce7ac85d831848ef7ce8f25aa010dd68bbf78b55e80201d" exitCode=2 Jul 06 20:58:40.779906 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.779847 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" event={"ID":"58229cdc-1038-4689-96c7-428db5647f04","Type":"ContainerDied","Data":"1b5ee6b8729d506ecce7ac85d831848ef7ce8f25aa010dd68bbf78b55e80201d"} Jul 06 20:58:40.781616 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.781588 2582 generic.go:356] "Generic (PLEG): container finished" podID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerID="500e0a710912fc4cd47ca9510e54a45fa79200dd842f4461643e41123eeaa87b" exitCode=2 Jul 06 20:58:40.781684 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.781633 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" event={"ID":"acc97a0d-30b2-4875-b3c6-931994dfca80","Type":"ContainerDied","Data":"500e0a710912fc4cd47ca9510e54a45fa79200dd842f4461643e41123eeaa87b"} Jul 06 20:58:40.810075 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:40.810005 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" podStartSLOduration=0.809991554 podStartE2EDuration="809.991554ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:58:40.806670491 +0000 UTC m=+1387.892225681" watchObservedRunningTime="2026-07-06 20:58:40.809991554 +0000 UTC m=+1387.895546747" Jul 06 20:58:41.788105 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:41.788072 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" event={"ID":"fc66437a-9311-48cd-963f-ac035c17b4a3","Type":"ContainerStarted","Data":"3fb74d3e299917e82fe3b78b7418cf0a420d26d52496318defad5ad32a609444"} Jul 06 20:58:41.788481 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:41.788307 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:41.788481 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:41.788359 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:41.789835 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:41.789810 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" Jul 06 20:58:41.810839 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:41.810798 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" podStartSLOduration=1.810784216 podStartE2EDuration="1.810784216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:58:41.809966 +0000 UTC m=+1388.895521192" watchObservedRunningTime="2026-07-06 20:58:41.810784216 +0000 UTC m=+1388.896339401" Jul 06 20:58:42.793181 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:42.793130 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" Jul 06 20:58:42.793644 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:42.793199 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:42.794490 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:42.794465 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Jul 06 20:58:43.442412 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:43.442354 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused" Jul 06 20:58:43.442839 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:43.442803 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused" Jul 06 20:58:43.800194 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:43.800161 2582 generic.go:356] "Generic (PLEG): container finished" podID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerID="f87d5b76137dc32aec4b9974792ead58976b523153e38c5e51ce50c6d0b7bdcc" exitCode=0 Jul 06 20:58:43.800509 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:43.800227 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" event={"ID":"acc97a0d-30b2-4875-b3c6-931994dfca80","Type":"ContainerDied","Data":"f87d5b76137dc32aec4b9974792ead58976b523153e38c5e51ce50c6d0b7bdcc"} Jul 06 20:58:43.800691 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:43.800658 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Jul 06 20:58:43.875596 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:43.875568 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:58:43.949378 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:43.949355 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:58:43.949509 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:43.949461 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc97a0d-30b2-4875-b3c6-931994dfca80-proxy-tls\") pod \"acc97a0d-30b2-4875-b3c6-931994dfca80\" (UID: \"acc97a0d-30b2-4875-b3c6-931994dfca80\") " Jul 06 20:58:43.949509 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:43.949502 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg6rx\" (UniqueName: \"kubernetes.io/projected/acc97a0d-30b2-4875-b3c6-931994dfca80-kube-api-access-mg6rx\") pod \"acc97a0d-30b2-4875-b3c6-931994dfca80\" (UID: \"acc97a0d-30b2-4875-b3c6-931994dfca80\") " Jul 06 20:58:43.949618 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:43.949539 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acc97a0d-30b2-4875-b3c6-931994dfca80-kube-rbac-proxy-sar-config\") pod \"acc97a0d-30b2-4875-b3c6-931994dfca80\" (UID: \"acc97a0d-30b2-4875-b3c6-931994dfca80\") " Jul 06 20:58:43.949947 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:43.949924 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc97a0d-30b2-4875-b3c6-931994dfca80-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "acc97a0d-30b2-4875-b3c6-931994dfca80" (UID: "acc97a0d-30b2-4875-b3c6-931994dfca80"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:58:43.951387 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:43.951367 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc97a0d-30b2-4875-b3c6-931994dfca80-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "acc97a0d-30b2-4875-b3c6-931994dfca80" (UID: "acc97a0d-30b2-4875-b3c6-931994dfca80"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:58:43.951477 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:43.951458 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc97a0d-30b2-4875-b3c6-931994dfca80-kube-api-access-mg6rx" (OuterVolumeSpecName: "kube-api-access-mg6rx") pod "acc97a0d-30b2-4875-b3c6-931994dfca80" (UID: "acc97a0d-30b2-4875-b3c6-931994dfca80"). InnerVolumeSpecName "kube-api-access-mg6rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:58:44.050605 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.050537 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58229cdc-1038-4689-96c7-428db5647f04-proxy-tls\") pod \"58229cdc-1038-4689-96c7-428db5647f04\" (UID: \"58229cdc-1038-4689-96c7-428db5647f04\") " Jul 06 20:58:44.050605 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.050604 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjr5l\" (UniqueName: \"kubernetes.io/projected/58229cdc-1038-4689-96c7-428db5647f04-kube-api-access-pjr5l\") pod \"58229cdc-1038-4689-96c7-428db5647f04\" (UID: \"58229cdc-1038-4689-96c7-428db5647f04\") " Jul 06 20:58:44.050744 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.050647 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58229cdc-1038-4689-96c7-428db5647f04-kube-rbac-proxy-sar-config\") pod \"58229cdc-1038-4689-96c7-428db5647f04\" (UID: \"58229cdc-1038-4689-96c7-428db5647f04\") " Jul 06 20:58:44.050809 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.050798 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acc97a0d-30b2-4875-b3c6-931994dfca80-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:58:44.050850 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.050814 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg6rx\" (UniqueName: \"kubernetes.io/projected/acc97a0d-30b2-4875-b3c6-931994dfca80-kube-api-access-mg6rx\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:58:44.050850 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.050823 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acc97a0d-30b2-4875-b3c6-931994dfca80-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:58:44.051070 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.051042 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58229cdc-1038-4689-96c7-428db5647f04-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58229cdc-1038-4689-96c7-428db5647f04" (UID: "58229cdc-1038-4689-96c7-428db5647f04"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:58:44.052480 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.052462 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58229cdc-1038-4689-96c7-428db5647f04-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58229cdc-1038-4689-96c7-428db5647f04" (UID: "58229cdc-1038-4689-96c7-428db5647f04"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:58:44.052552 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.052500 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58229cdc-1038-4689-96c7-428db5647f04-kube-api-access-pjr5l" (OuterVolumeSpecName: "kube-api-access-pjr5l") pod "58229cdc-1038-4689-96c7-428db5647f04" (UID: "58229cdc-1038-4689-96c7-428db5647f04"). InnerVolumeSpecName "kube-api-access-pjr5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:58:44.151822 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.151792 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58229cdc-1038-4689-96c7-428db5647f04-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:58:44.151822 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.151815 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjr5l\" (UniqueName: \"kubernetes.io/projected/58229cdc-1038-4689-96c7-428db5647f04-kube-api-access-pjr5l\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:58:44.151822 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.151824 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58229cdc-1038-4689-96c7-428db5647f04-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 20:58:44.805707 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.805673 2582 generic.go:356] "Generic (PLEG): container finished" podID="58229cdc-1038-4689-96c7-428db5647f04" containerID="eb81fb4dcf052ba79486cb99b0a86d8cbb8974039da65afed41d607cd99b9ca3" exitCode=0 Jul 06 20:58:44.806173 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.805730 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" Jul 06 20:58:44.806173 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.805747 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" event={"ID":"58229cdc-1038-4689-96c7-428db5647f04","Type":"ContainerDied","Data":"eb81fb4dcf052ba79486cb99b0a86d8cbb8974039da65afed41d607cd99b9ca3"} Jul 06 20:58:44.806173 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.805825 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw" event={"ID":"58229cdc-1038-4689-96c7-428db5647f04","Type":"ContainerDied","Data":"5aecd9597775d44ad0d6e7c98c1b49c3f5539428feb1a8d0bd98c14c5e779be3"} Jul 06 20:58:44.806173 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.805862 2582 scope.go:117] "RemoveContainer" containerID="1b5ee6b8729d506ecce7ac85d831848ef7ce8f25aa010dd68bbf78b55e80201d" Jul 06 20:58:44.807413 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.807397 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" Jul 06 20:58:44.807413 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.807400 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57" event={"ID":"acc97a0d-30b2-4875-b3c6-931994dfca80","Type":"ContainerDied","Data":"7961a4288248fc344e6c97383592e6b2b6610d1c8d7250aaa854a4398348f774"} Jul 06 20:58:44.815483 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.815467 2582 scope.go:117] "RemoveContainer" containerID="eb81fb4dcf052ba79486cb99b0a86d8cbb8974039da65afed41d607cd99b9ca3" Jul 06 20:58:44.823078 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.823062 2582 scope.go:117] "RemoveContainer" containerID="1b5ee6b8729d506ecce7ac85d831848ef7ce8f25aa010dd68bbf78b55e80201d" Jul 06 20:58:44.823340 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:58:44.823322 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5ee6b8729d506ecce7ac85d831848ef7ce8f25aa010dd68bbf78b55e80201d\": container with ID starting with 1b5ee6b8729d506ecce7ac85d831848ef7ce8f25aa010dd68bbf78b55e80201d not found: ID does not exist" containerID="1b5ee6b8729d506ecce7ac85d831848ef7ce8f25aa010dd68bbf78b55e80201d" Jul 06 20:58:44.823386 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.823349 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5ee6b8729d506ecce7ac85d831848ef7ce8f25aa010dd68bbf78b55e80201d"} err="failed to get container status \"1b5ee6b8729d506ecce7ac85d831848ef7ce8f25aa010dd68bbf78b55e80201d\": rpc error: code = NotFound desc = could not find container \"1b5ee6b8729d506ecce7ac85d831848ef7ce8f25aa010dd68bbf78b55e80201d\": container with ID starting with 1b5ee6b8729d506ecce7ac85d831848ef7ce8f25aa010dd68bbf78b55e80201d not found: ID does not exist" Jul 06 20:58:44.823386 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.823377 2582 scope.go:117] "RemoveContainer" containerID="eb81fb4dcf052ba79486cb99b0a86d8cbb8974039da65afed41d607cd99b9ca3" Jul 06 20:58:44.823611 ip-10-0-142-142 kubenswrapper[2582]: E0706 20:58:44.823595 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb81fb4dcf052ba79486cb99b0a86d8cbb8974039da65afed41d607cd99b9ca3\": container with ID starting with eb81fb4dcf052ba79486cb99b0a86d8cbb8974039da65afed41d607cd99b9ca3 not found: ID does not exist" containerID="eb81fb4dcf052ba79486cb99b0a86d8cbb8974039da65afed41d607cd99b9ca3" Jul 06 20:58:44.823643 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.823617 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb81fb4dcf052ba79486cb99b0a86d8cbb8974039da65afed41d607cd99b9ca3"} err="failed to get container status \"eb81fb4dcf052ba79486cb99b0a86d8cbb8974039da65afed41d607cd99b9ca3\": rpc error: code = NotFound desc = could not find container \"eb81fb4dcf052ba79486cb99b0a86d8cbb8974039da65afed41d607cd99b9ca3\": container with ID starting with eb81fb4dcf052ba79486cb99b0a86d8cbb8974039da65afed41d607cd99b9ca3 not found: ID does not exist" Jul 06 20:58:44.823643 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.823631 2582 scope.go:117] "RemoveContainer" containerID="500e0a710912fc4cd47ca9510e54a45fa79200dd842f4461643e41123eeaa87b" Jul 06 20:58:44.831043 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.831023 2582 scope.go:117] "RemoveContainer" containerID="f87d5b76137dc32aec4b9974792ead58976b523153e38c5e51ce50c6d0b7bdcc" Jul 06 20:58:44.846641 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.846621 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57"] Jul 06 20:58:44.856079 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.856058 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-24ddf-predictor-b8f9d9964-25s57"] Jul 06 20:58:44.878415 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.878397 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw"] Jul 06 20:58:44.886541 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:44.886516 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-24ddf-predictor-665cbb8c8f-25wjw"] Jul 06 20:58:45.538530 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:45.538492 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58229cdc-1038-4689-96c7-428db5647f04" path="/var/lib/kubelet/pods/58229cdc-1038-4689-96c7-428db5647f04/volumes" Jul 06 20:58:45.539147 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:45.539125 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" path="/var/lib/kubelet/pods/acc97a0d-30b2-4875-b3c6-931994dfca80/volumes" Jul 06 20:58:47.646115 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:47.646075 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused" Jul 06 20:58:47.797523 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:47.797491 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:58:47.798061 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:47.798031 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" Jul 06 20:58:48.652048 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:48.652006 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" Jul 06 20:58:48.804957 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:48.804929 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 20:58:48.805421 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:48.805391 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Jul 06 20:58:57.646500 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:57.646466 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 20:58:57.798252 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:57.798217 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" Jul 06 20:58:58.652600 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:58.652575 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 20:58:58.806258 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:58:58.806221 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Jul 06 20:59:07.798751 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:59:07.798709 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" Jul 06 20:59:08.806060 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:59:08.806024 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Jul 06 20:59:17.798901 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:59:17.798864 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" Jul 06 20:59:18.806034 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:59:18.805998 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Jul 06 20:59:27.798962 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:59:27.798886 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 20:59:28.806553 ip-10-0-142-142 kubenswrapper[2582]: I0706 20:59:28.806527 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 21:00:33.454945 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:00:33.454902 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 21:00:33.459253 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:00:33.459234 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 21:05:33.481040 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:05:33.481015 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 21:05:33.485665 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:05:33.485645 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 21:07:25.136932 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.136849 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m"] Jul 06 21:07:25.137368 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.137228 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kserve-container" Jul 06 21:07:25.137368 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.137239 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kserve-container" Jul 06 21:07:25.137368 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.137254 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kube-rbac-proxy" Jul 06 21:07:25.137368 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.137259 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kube-rbac-proxy" Jul 06 21:07:25.137368 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.137266 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kube-rbac-proxy" Jul 06 21:07:25.137368 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.137272 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kube-rbac-proxy" Jul 06 21:07:25.137368 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.137286 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kserve-container" Jul 06 21:07:25.137368 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.137290 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kserve-container" Jul 06 21:07:25.137368 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.137345 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kube-rbac-proxy" Jul 06 21:07:25.137368 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.137352 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kserve-container" Jul 06 21:07:25.137368 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.137361 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="acc97a0d-30b2-4875-b3c6-931994dfca80" containerName="kserve-container" Jul 06 21:07:25.137368 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.137367 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="58229cdc-1038-4689-96c7-428db5647f04" containerName="kube-rbac-proxy" Jul 06 21:07:25.140490 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.140473 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:25.143611 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.143586 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt"] Jul 06 21:07:25.143846 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.143793 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kserve-container" containerID="cri-o://6d8e6e7f9b25fe0e41c4ed7a922c6f7e486454f638ab7f3231e79a7546eabd9c" gracePeriod=30 Jul 06 21:07:25.143925 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.143830 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kube-rbac-proxy" containerID="cri-o://a0bc502a26f272992c15d7e6e95cb4fc91e85484ba72363067b05e24fbe7a3ed" gracePeriod=30 Jul 06 21:07:25.143925 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.143907 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-999e1-kube-rbac-proxy-sar-config\"" Jul 06 21:07:25.144088 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.144068 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-999e1-predictor-serving-cert\"" Jul 06 21:07:25.157734 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.157714 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m"] Jul 06 21:07:25.243723 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.243691 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw"] Jul 06 21:07:25.247195 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.247174 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:25.247528 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.247460 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ndn4\" (UniqueName: \"kubernetes.io/projected/fae36521-e6ea-446b-892c-51cc9ec16d81-kube-api-access-4ndn4\") pod \"success-200-isvc-999e1-predictor-57944785f4-q888m\" (UID: \"fae36521-e6ea-446b-892c-51cc9ec16d81\") " pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:25.247655 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.247560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fae36521-e6ea-446b-892c-51cc9ec16d81-proxy-tls\") pod \"success-200-isvc-999e1-predictor-57944785f4-q888m\" (UID: \"fae36521-e6ea-446b-892c-51cc9ec16d81\") " pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:25.247655 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.247614 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fae36521-e6ea-446b-892c-51cc9ec16d81-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-999e1-predictor-57944785f4-q888m\" (UID: \"fae36521-e6ea-446b-892c-51cc9ec16d81\") " pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:25.250084 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.250064 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-999e1-predictor-serving-cert\"" Jul 06 21:07:25.250187 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.250160 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-999e1-kube-rbac-proxy-sar-config\"" Jul 06 21:07:25.266095 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.266068 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw"] Jul 06 21:07:25.348137 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.348108 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ndn4\" (UniqueName: \"kubernetes.io/projected/fae36521-e6ea-446b-892c-51cc9ec16d81-kube-api-access-4ndn4\") pod \"success-200-isvc-999e1-predictor-57944785f4-q888m\" (UID: \"fae36521-e6ea-446b-892c-51cc9ec16d81\") " pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:25.348306 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.348154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46b44663-269b-4ead-bbd4-09f1909c4e9b-proxy-tls\") pod \"error-404-isvc-999e1-predictor-fd6f8745f-njpnw\" (UID: \"46b44663-269b-4ead-bbd4-09f1909c4e9b\") " pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:25.348306 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.348189 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fae36521-e6ea-446b-892c-51cc9ec16d81-proxy-tls\") pod \"success-200-isvc-999e1-predictor-57944785f4-q888m\" (UID: \"fae36521-e6ea-446b-892c-51cc9ec16d81\") " pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:25.348306 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.348206 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46b44663-269b-4ead-bbd4-09f1909c4e9b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-999e1-predictor-fd6f8745f-njpnw\" (UID: \"46b44663-269b-4ead-bbd4-09f1909c4e9b\") " pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:25.348306 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.348234 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fae36521-e6ea-446b-892c-51cc9ec16d81-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-999e1-predictor-57944785f4-q888m\" (UID: \"fae36521-e6ea-446b-892c-51cc9ec16d81\") " pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:25.348306 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.348261 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk5p2\" (UniqueName: \"kubernetes.io/projected/46b44663-269b-4ead-bbd4-09f1909c4e9b-kube-api-access-wk5p2\") pod \"error-404-isvc-999e1-predictor-fd6f8745f-njpnw\" (UID: \"46b44663-269b-4ead-bbd4-09f1909c4e9b\") " pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:25.348853 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.348834 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fae36521-e6ea-446b-892c-51cc9ec16d81-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-999e1-predictor-57944785f4-q888m\" (UID: \"fae36521-e6ea-446b-892c-51cc9ec16d81\") " pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:25.350500 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.350477 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fae36521-e6ea-446b-892c-51cc9ec16d81-proxy-tls\") pod \"success-200-isvc-999e1-predictor-57944785f4-q888m\" (UID: \"fae36521-e6ea-446b-892c-51cc9ec16d81\") " pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:25.363671 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.363646 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq"] Jul 06 21:07:25.363908 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.363882 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kserve-container" containerID="cri-o://45f5835189f3369d4e17a6afea56461d74ec2684aeeb2d776261a7a6669bdeb5" gracePeriod=30 Jul 06 21:07:25.363975 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.363899 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kube-rbac-proxy" containerID="cri-o://8e172d172a02a635c50fe9d40139cb33b8f01019648088f3ddd74b0ecf4c16e7" gracePeriod=30 Jul 06 21:07:25.380665 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.380623 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ndn4\" (UniqueName: \"kubernetes.io/projected/fae36521-e6ea-446b-892c-51cc9ec16d81-kube-api-access-4ndn4\") pod \"success-200-isvc-999e1-predictor-57944785f4-q888m\" (UID: \"fae36521-e6ea-446b-892c-51cc9ec16d81\") " pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:25.449620 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.449542 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46b44663-269b-4ead-bbd4-09f1909c4e9b-proxy-tls\") pod \"error-404-isvc-999e1-predictor-fd6f8745f-njpnw\" (UID: \"46b44663-269b-4ead-bbd4-09f1909c4e9b\") " pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:25.449620 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.449587 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46b44663-269b-4ead-bbd4-09f1909c4e9b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-999e1-predictor-fd6f8745f-njpnw\" (UID: \"46b44663-269b-4ead-bbd4-09f1909c4e9b\") " pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:25.449837 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.449634 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk5p2\" (UniqueName: \"kubernetes.io/projected/46b44663-269b-4ead-bbd4-09f1909c4e9b-kube-api-access-wk5p2\") pod \"error-404-isvc-999e1-predictor-fd6f8745f-njpnw\" (UID: \"46b44663-269b-4ead-bbd4-09f1909c4e9b\") " pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:25.450294 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.450275 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46b44663-269b-4ead-bbd4-09f1909c4e9b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-999e1-predictor-fd6f8745f-njpnw\" (UID: \"46b44663-269b-4ead-bbd4-09f1909c4e9b\") " pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:25.451876 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.451857 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46b44663-269b-4ead-bbd4-09f1909c4e9b-proxy-tls\") pod \"error-404-isvc-999e1-predictor-fd6f8745f-njpnw\" (UID: \"46b44663-269b-4ead-bbd4-09f1909c4e9b\") " pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:25.460228 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.460208 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:25.465768 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.465738 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk5p2\" (UniqueName: \"kubernetes.io/projected/46b44663-269b-4ead-bbd4-09f1909c4e9b-kube-api-access-wk5p2\") pod \"error-404-isvc-999e1-predictor-fd6f8745f-njpnw\" (UID: \"46b44663-269b-4ead-bbd4-09f1909c4e9b\") " pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:25.557700 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.557666 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:25.600691 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.600655 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m"] Jul 06 21:07:25.602824 ip-10-0-142-142 kubenswrapper[2582]: W0706 21:07:25.602789 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae36521_e6ea_446b_892c_51cc9ec16d81.slice/crio-3f5f016225c7eaadd97d82d536ef4269c3889dd864069ee317de0e736b4ff4c8 WatchSource:0}: Error finding container 3f5f016225c7eaadd97d82d536ef4269c3889dd864069ee317de0e736b4ff4c8: Status 404 returned error can't find the container with id 3f5f016225c7eaadd97d82d536ef4269c3889dd864069ee317de0e736b4ff4c8 Jul 06 21:07:25.702536 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:25.702517 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw"] Jul 06 21:07:25.705851 ip-10-0-142-142 kubenswrapper[2582]: W0706 21:07:25.705826 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b44663_269b_4ead_bbd4_09f1909c4e9b.slice/crio-872bd080b8be5eb42a5493154747358081e45d6e7f269d813d7bc3c48b43fc95 WatchSource:0}: Error finding container 872bd080b8be5eb42a5493154747358081e45d6e7f269d813d7bc3c48b43fc95: Status 404 returned error can't find the container with id 872bd080b8be5eb42a5493154747358081e45d6e7f269d813d7bc3c48b43fc95 Jul 06 21:07:26.179748 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.179714 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" event={"ID":"46b44663-269b-4ead-bbd4-09f1909c4e9b","Type":"ContainerStarted","Data":"88dfc3232725fe4912434fb592c3a648ccb8e6641a4153521029da3759d18c73"} Jul 06 21:07:26.180168 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.179795 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" event={"ID":"46b44663-269b-4ead-bbd4-09f1909c4e9b","Type":"ContainerStarted","Data":"7b758bbeca9ff940fcf2798cd30da526da54a7d5ecf2a8902050781ddc87e1ca"} Jul 06 21:07:26.180168 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.179822 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:26.180168 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.179840 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" event={"ID":"46b44663-269b-4ead-bbd4-09f1909c4e9b","Type":"ContainerStarted","Data":"872bd080b8be5eb42a5493154747358081e45d6e7f269d813d7bc3c48b43fc95"} Jul 06 21:07:26.187401 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.187218 2582 generic.go:356] "Generic (PLEG): container finished" podID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerID="8e172d172a02a635c50fe9d40139cb33b8f01019648088f3ddd74b0ecf4c16e7" exitCode=2 Jul 06 21:07:26.187401 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.187382 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" event={"ID":"290079df-dd51-4baa-8a85-08d49d7fcf25","Type":"ContainerDied","Data":"8e172d172a02a635c50fe9d40139cb33b8f01019648088f3ddd74b0ecf4c16e7"} Jul 06 21:07:26.189379 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.189358 2582 generic.go:356] "Generic (PLEG): container finished" podID="0353baf0-e073-4f81-aa0c-68e01997550c" containerID="a0bc502a26f272992c15d7e6e95cb4fc91e85484ba72363067b05e24fbe7a3ed" exitCode=2 Jul 06 21:07:26.189519 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.189432 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" event={"ID":"0353baf0-e073-4f81-aa0c-68e01997550c","Type":"ContainerDied","Data":"a0bc502a26f272992c15d7e6e95cb4fc91e85484ba72363067b05e24fbe7a3ed"} Jul 06 21:07:26.191014 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.190996 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" event={"ID":"fae36521-e6ea-446b-892c-51cc9ec16d81","Type":"ContainerStarted","Data":"1faa1f69cd65da5737e8553069b40a75b0fdad7ba50d8460e0efb85b908d67cb"} Jul 06 21:07:26.191120 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.191042 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" event={"ID":"fae36521-e6ea-446b-892c-51cc9ec16d81","Type":"ContainerStarted","Data":"762e86aa0c0adde717094bca646286124858e1f9e5b1b8853ce40b887049914d"} Jul 06 21:07:26.191120 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.191073 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" event={"ID":"fae36521-e6ea-446b-892c-51cc9ec16d81","Type":"ContainerStarted","Data":"3f5f016225c7eaadd97d82d536ef4269c3889dd864069ee317de0e736b4ff4c8"} Jul 06 21:07:26.191120 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.191107 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:26.191266 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.191157 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:26.192463 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.192441 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused" Jul 06 21:07:26.205045 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.205001 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" podStartSLOduration=1.204988743 podStartE2EDuration="1.204988743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:07:26.20422063 +0000 UTC m=+1913.289775823" watchObservedRunningTime="2026-07-06 21:07:26.204988743 +0000 UTC m=+1913.290543914" Jul 06 21:07:26.230239 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:26.230194 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" podStartSLOduration=1.230180329 podStartE2EDuration="1.230180329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:07:26.229149675 +0000 UTC m=+1913.314704879" watchObservedRunningTime="2026-07-06 21:07:26.230180329 +0000 UTC m=+1913.315735521" Jul 06 21:07:27.195639 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:27.195605 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:27.196049 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:27.195668 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused" Jul 06 21:07:27.196893 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:27.196871 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused" Jul 06 21:07:27.640476 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:27.640432 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused" Jul 06 21:07:27.646032 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:27.645996 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused" Jul 06 21:07:28.201645 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.201618 2582 generic.go:356] "Generic (PLEG): container finished" podID="0353baf0-e073-4f81-aa0c-68e01997550c" containerID="6d8e6e7f9b25fe0e41c4ed7a922c6f7e486454f638ab7f3231e79a7546eabd9c" exitCode=0 Jul 06 21:07:28.201994 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.201689 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" event={"ID":"0353baf0-e073-4f81-aa0c-68e01997550c","Type":"ContainerDied","Data":"6d8e6e7f9b25fe0e41c4ed7a922c6f7e486454f638ab7f3231e79a7546eabd9c"} Jul 06 21:07:28.202068 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.202050 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused" Jul 06 21:07:28.292028 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.292005 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 21:07:28.374223 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.374194 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0353baf0-e073-4f81-aa0c-68e01997550c-kube-rbac-proxy-sar-config\") pod \"0353baf0-e073-4f81-aa0c-68e01997550c\" (UID: \"0353baf0-e073-4f81-aa0c-68e01997550c\") " Jul 06 21:07:28.374363 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.374260 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0353baf0-e073-4f81-aa0c-68e01997550c-proxy-tls\") pod \"0353baf0-e073-4f81-aa0c-68e01997550c\" (UID: \"0353baf0-e073-4f81-aa0c-68e01997550c\") " Jul 06 21:07:28.374413 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.374361 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8ff2\" (UniqueName: \"kubernetes.io/projected/0353baf0-e073-4f81-aa0c-68e01997550c-kube-api-access-l8ff2\") pod \"0353baf0-e073-4f81-aa0c-68e01997550c\" (UID: \"0353baf0-e073-4f81-aa0c-68e01997550c\") " Jul 06 21:07:28.374580 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.374559 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0353baf0-e073-4f81-aa0c-68e01997550c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0353baf0-e073-4f81-aa0c-68e01997550c" (UID: "0353baf0-e073-4f81-aa0c-68e01997550c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:07:28.374668 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.374655 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0353baf0-e073-4f81-aa0c-68e01997550c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:07:28.376154 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.376130 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0353baf0-e073-4f81-aa0c-68e01997550c-kube-api-access-l8ff2" (OuterVolumeSpecName: "kube-api-access-l8ff2") pod "0353baf0-e073-4f81-aa0c-68e01997550c" (UID: "0353baf0-e073-4f81-aa0c-68e01997550c"). InnerVolumeSpecName "kube-api-access-l8ff2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:07:28.376252 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.376237 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0353baf0-e073-4f81-aa0c-68e01997550c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0353baf0-e073-4f81-aa0c-68e01997550c" (UID: "0353baf0-e073-4f81-aa0c-68e01997550c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:07:28.475718 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.475664 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8ff2\" (UniqueName: \"kubernetes.io/projected/0353baf0-e073-4f81-aa0c-68e01997550c-kube-api-access-l8ff2\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:07:28.475718 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.475684 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0353baf0-e073-4f81-aa0c-68e01997550c-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:07:28.699822 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.699793 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 21:07:28.778906 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.778824 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/290079df-dd51-4baa-8a85-08d49d7fcf25-kube-rbac-proxy-sar-config\") pod \"290079df-dd51-4baa-8a85-08d49d7fcf25\" (UID: \"290079df-dd51-4baa-8a85-08d49d7fcf25\") " Jul 06 21:07:28.778906 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.778895 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/290079df-dd51-4baa-8a85-08d49d7fcf25-proxy-tls\") pod \"290079df-dd51-4baa-8a85-08d49d7fcf25\" (UID: \"290079df-dd51-4baa-8a85-08d49d7fcf25\") " Jul 06 21:07:28.779091 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.778921 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsg8p\" (UniqueName: \"kubernetes.io/projected/290079df-dd51-4baa-8a85-08d49d7fcf25-kube-api-access-gsg8p\") pod \"290079df-dd51-4baa-8a85-08d49d7fcf25\" (UID: \"290079df-dd51-4baa-8a85-08d49d7fcf25\") " Jul 06 21:07:28.779212 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.779188 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290079df-dd51-4baa-8a85-08d49d7fcf25-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "290079df-dd51-4baa-8a85-08d49d7fcf25" (UID: "290079df-dd51-4baa-8a85-08d49d7fcf25"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:07:28.780978 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.780947 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290079df-dd51-4baa-8a85-08d49d7fcf25-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "290079df-dd51-4baa-8a85-08d49d7fcf25" (UID: "290079df-dd51-4baa-8a85-08d49d7fcf25"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:07:28.781101 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.781060 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290079df-dd51-4baa-8a85-08d49d7fcf25-kube-api-access-gsg8p" (OuterVolumeSpecName: "kube-api-access-gsg8p") pod "290079df-dd51-4baa-8a85-08d49d7fcf25" (UID: "290079df-dd51-4baa-8a85-08d49d7fcf25"). InnerVolumeSpecName "kube-api-access-gsg8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:07:28.879654 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.879627 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/290079df-dd51-4baa-8a85-08d49d7fcf25-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:07:28.879654 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.879650 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/290079df-dd51-4baa-8a85-08d49d7fcf25-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:07:28.879654 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:28.879660 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsg8p\" (UniqueName: \"kubernetes.io/projected/290079df-dd51-4baa-8a85-08d49d7fcf25-kube-api-access-gsg8p\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:07:29.208411 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.208375 2582 generic.go:356] "Generic (PLEG): container finished" podID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerID="45f5835189f3369d4e17a6afea56461d74ec2684aeeb2d776261a7a6669bdeb5" exitCode=0 Jul 06 21:07:29.208880 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.208442 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" Jul 06 21:07:29.208880 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.208460 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" event={"ID":"290079df-dd51-4baa-8a85-08d49d7fcf25","Type":"ContainerDied","Data":"45f5835189f3369d4e17a6afea56461d74ec2684aeeb2d776261a7a6669bdeb5"} Jul 06 21:07:29.208880 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.208521 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" event={"ID":"290079df-dd51-4baa-8a85-08d49d7fcf25","Type":"ContainerDied","Data":"2c522ad44f7a6d77584423698b3e26e3bc75185e09939a276f3fbebef4134d42"} Jul 06 21:07:29.208880 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.208552 2582 scope.go:117] "RemoveContainer" containerID="8e172d172a02a635c50fe9d40139cb33b8f01019648088f3ddd74b0ecf4c16e7" Jul 06 21:07:29.210222 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.210201 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" Jul 06 21:07:29.210356 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.210203 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt" event={"ID":"0353baf0-e073-4f81-aa0c-68e01997550c","Type":"ContainerDied","Data":"bc81b8c3ad12dc1fbcd44867dc543cac76e0e49f3adae8e57531be2cc4445112"} Jul 06 21:07:29.217970 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.217953 2582 scope.go:117] "RemoveContainer" containerID="45f5835189f3369d4e17a6afea56461d74ec2684aeeb2d776261a7a6669bdeb5" Jul 06 21:07:29.225550 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.225535 2582 scope.go:117] "RemoveContainer" containerID="8e172d172a02a635c50fe9d40139cb33b8f01019648088f3ddd74b0ecf4c16e7" Jul 06 21:07:29.225804 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:07:29.225786 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e172d172a02a635c50fe9d40139cb33b8f01019648088f3ddd74b0ecf4c16e7\": container with ID starting with 8e172d172a02a635c50fe9d40139cb33b8f01019648088f3ddd74b0ecf4c16e7 not found: ID does not exist" containerID="8e172d172a02a635c50fe9d40139cb33b8f01019648088f3ddd74b0ecf4c16e7" Jul 06 21:07:29.225873 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.225811 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e172d172a02a635c50fe9d40139cb33b8f01019648088f3ddd74b0ecf4c16e7"} err="failed to get container status \"8e172d172a02a635c50fe9d40139cb33b8f01019648088f3ddd74b0ecf4c16e7\": rpc error: code = NotFound desc = could not find container \"8e172d172a02a635c50fe9d40139cb33b8f01019648088f3ddd74b0ecf4c16e7\": container with ID starting with 8e172d172a02a635c50fe9d40139cb33b8f01019648088f3ddd74b0ecf4c16e7 not found: ID does not exist" Jul 06 21:07:29.225873 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.225827 2582 scope.go:117] "RemoveContainer" containerID="45f5835189f3369d4e17a6afea56461d74ec2684aeeb2d776261a7a6669bdeb5" Jul 06 21:07:29.226030 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:07:29.226011 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45f5835189f3369d4e17a6afea56461d74ec2684aeeb2d776261a7a6669bdeb5\": container with ID starting with 45f5835189f3369d4e17a6afea56461d74ec2684aeeb2d776261a7a6669bdeb5 not found: ID does not exist" containerID="45f5835189f3369d4e17a6afea56461d74ec2684aeeb2d776261a7a6669bdeb5" Jul 06 21:07:29.226081 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.226036 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f5835189f3369d4e17a6afea56461d74ec2684aeeb2d776261a7a6669bdeb5"} err="failed to get container status \"45f5835189f3369d4e17a6afea56461d74ec2684aeeb2d776261a7a6669bdeb5\": rpc error: code = NotFound desc = could not find container \"45f5835189f3369d4e17a6afea56461d74ec2684aeeb2d776261a7a6669bdeb5\": container with ID starting with 45f5835189f3369d4e17a6afea56461d74ec2684aeeb2d776261a7a6669bdeb5 not found: ID does not exist" Jul 06 21:07:29.226081 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.226051 2582 scope.go:117] "RemoveContainer" containerID="a0bc502a26f272992c15d7e6e95cb4fc91e85484ba72363067b05e24fbe7a3ed" Jul 06 21:07:29.232670 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.232656 2582 scope.go:117] "RemoveContainer" containerID="6d8e6e7f9b25fe0e41c4ed7a922c6f7e486454f638ab7f3231e79a7546eabd9c" Jul 06 21:07:29.238268 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.238245 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq"] Jul 06 21:07:29.245101 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.245076 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq"] Jul 06 21:07:29.258717 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.258692 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt"] Jul 06 21:07:29.264367 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.264347 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-858c3-predictor-845dd5dbd6-f4bmt"] Jul 06 21:07:29.538902 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.538808 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" path="/var/lib/kubelet/pods/0353baf0-e073-4f81-aa0c-68e01997550c/volumes" Jul 06 21:07:29.539345 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.539323 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" path="/var/lib/kubelet/pods/290079df-dd51-4baa-8a85-08d49d7fcf25/volumes" Jul 06 21:07:29.647644 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.647612 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": context deadline exceeded" Jul 06 21:07:29.652924 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:29.652898 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-858c3-predictor-6b7b999659-kmlhq" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: i/o timeout" Jul 06 21:07:32.200418 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:32.200387 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:07:32.201076 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:32.201047 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused" Jul 06 21:07:33.206877 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:33.206843 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:07:33.207464 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:33.207435 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused" Jul 06 21:07:42.201947 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:42.201904 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused" Jul 06 21:07:43.207709 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:43.207675 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused" Jul 06 21:07:52.201546 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:52.201507 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused" Jul 06 21:07:53.207604 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:53.207543 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused" Jul 06 21:07:54.974880 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.974847 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2"] Jul 06 21:07:54.975243 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.975206 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kserve-container" Jul 06 21:07:54.975243 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.975217 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kserve-container" Jul 06 21:07:54.975243 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.975224 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kube-rbac-proxy" Jul 06 21:07:54.975243 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.975231 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kube-rbac-proxy" Jul 06 21:07:54.975243 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.975239 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kserve-container" Jul 06 21:07:54.975243 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.975245 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kserve-container" Jul 06 21:07:54.975431 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.975253 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kube-rbac-proxy" Jul 06 21:07:54.975431 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.975258 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kube-rbac-proxy" Jul 06 21:07:54.975431 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.975308 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kube-rbac-proxy" Jul 06 21:07:54.975431 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.975314 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="290079df-dd51-4baa-8a85-08d49d7fcf25" containerName="kserve-container" Jul 06 21:07:54.975431 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.975320 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kserve-container" Jul 06 21:07:54.975431 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.975325 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0353baf0-e073-4f81-aa0c-68e01997550c" containerName="kube-rbac-proxy" Jul 06 21:07:54.978417 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.978399 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:07:54.986090 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.986073 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-32e32-predictor-serving-cert\"" Jul 06 21:07:54.986820 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.986805 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-32e32-kube-rbac-proxy-sar-config\"" Jul 06 21:07:54.994621 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:54.994602 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2"] Jul 06 21:07:55.035905 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.035870 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt"] Jul 06 21:07:55.036112 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.036086 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kserve-container" containerID="cri-o://3d361b7695f739d4ac42643c2ed06aab46519d1e90756b4b494c6c6603ebb4ef" gracePeriod=30 Jul 06 21:07:55.036210 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.036099 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kube-rbac-proxy" containerID="cri-o://6cc3b399501e28eeada8f6c90eb1c0972cd9efd7e607530dbf78d133f7732bcd" gracePeriod=30 Jul 06 21:07:55.095276 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.095239 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2\" (UID: \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\") " pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:07:55.095276 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.095279 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-proxy-tls\") pod \"success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2\" (UID: \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\") " pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:07:55.095479 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.095353 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5bm8\" (UniqueName: \"kubernetes.io/projected/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-kube-api-access-k5bm8\") pod \"success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2\" (UID: \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\") " pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:07:55.153972 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.153940 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf"] Jul 06 21:07:55.157470 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.157446 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:07:55.160689 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.160668 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-32e32-predictor-serving-cert\"" Jul 06 21:07:55.160890 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.160873 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-32e32-kube-rbac-proxy-sar-config\"" Jul 06 21:07:55.173355 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.173331 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf"] Jul 06 21:07:55.196290 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.196257 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5bm8\" (UniqueName: \"kubernetes.io/projected/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-kube-api-access-k5bm8\") pod \"success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2\" (UID: \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\") " pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:07:55.196437 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.196387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2\" (UID: \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\") " pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:07:55.196437 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.196413 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-proxy-tls\") pod \"success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2\" (UID: \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\") " pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:07:55.197113 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.197088 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2\" (UID: \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\") " pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:07:55.198689 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.198663 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-proxy-tls\") pod \"success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2\" (UID: \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\") " pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:07:55.205510 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.205485 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv"] Jul 06 21:07:55.205801 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.205741 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kserve-container" containerID="cri-o://6d87cf3a52332597f96a1bd6e76ca5a4b8b1854f7f0b4f3afde9762a5dd8db9a" gracePeriod=30 Jul 06 21:07:55.205968 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.205853 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kube-rbac-proxy" containerID="cri-o://3fb74d3e299917e82fe3b78b7418cf0a420d26d52496318defad5ad32a609444" gracePeriod=30 Jul 06 21:07:55.211717 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.211700 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5bm8\" (UniqueName: \"kubernetes.io/projected/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-kube-api-access-k5bm8\") pod \"success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2\" (UID: \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\") " pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:07:55.288772 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.288693 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:07:55.296972 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.296945 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x6kr\" (UniqueName: \"kubernetes.io/projected/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-kube-api-access-9x6kr\") pod \"error-404-isvc-32e32-predictor-98848c587-cfphf\" (UID: \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\") " pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:07:55.297109 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.297001 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-32e32-predictor-98848c587-cfphf\" (UID: \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\") " pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:07:55.297173 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.297098 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-proxy-tls\") pod \"error-404-isvc-32e32-predictor-98848c587-cfphf\" (UID: \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\") " pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:07:55.330556 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.330529 2582 generic.go:356] "Generic (PLEG): container finished" podID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerID="6cc3b399501e28eeada8f6c90eb1c0972cd9efd7e607530dbf78d133f7732bcd" exitCode=2 Jul 06 21:07:55.330658 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.330606 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" event={"ID":"6611d492-131b-4cb5-81e2-e67702aac1f2","Type":"ContainerDied","Data":"6cc3b399501e28eeada8f6c90eb1c0972cd9efd7e607530dbf78d133f7732bcd"} Jul 06 21:07:55.332372 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.332351 2582 generic.go:356] "Generic (PLEG): container finished" podID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerID="3fb74d3e299917e82fe3b78b7418cf0a420d26d52496318defad5ad32a609444" exitCode=2 Jul 06 21:07:55.332459 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.332382 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" event={"ID":"fc66437a-9311-48cd-963f-ac035c17b4a3","Type":"ContainerDied","Data":"3fb74d3e299917e82fe3b78b7418cf0a420d26d52496318defad5ad32a609444"} Jul 06 21:07:55.398419 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.398389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-proxy-tls\") pod \"error-404-isvc-32e32-predictor-98848c587-cfphf\" (UID: \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\") " pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:07:55.398574 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.398437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x6kr\" (UniqueName: \"kubernetes.io/projected/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-kube-api-access-9x6kr\") pod \"error-404-isvc-32e32-predictor-98848c587-cfphf\" (UID: \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\") " pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:07:55.398574 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.398465 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-32e32-predictor-98848c587-cfphf\" (UID: \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\") " pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:07:55.399203 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.399178 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-32e32-predictor-98848c587-cfphf\" (UID: \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\") " pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:07:55.401052 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.401027 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-proxy-tls\") pod \"error-404-isvc-32e32-predictor-98848c587-cfphf\" (UID: \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\") " pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:07:55.408785 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.408750 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x6kr\" (UniqueName: \"kubernetes.io/projected/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-kube-api-access-9x6kr\") pod \"error-404-isvc-32e32-predictor-98848c587-cfphf\" (UID: \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\") " pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:07:55.429597 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.429575 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2"] Jul 06 21:07:55.430853 ip-10-0-142-142 kubenswrapper[2582]: W0706 21:07:55.430825 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e3ebfc_8b2e_4020_b04c_6f294c74ac4e.slice/crio-1f5366190fe17fadfab59fa0c09b5848987bf0d46c5555e06348c3549b53eb2d WatchSource:0}: Error finding container 1f5366190fe17fadfab59fa0c09b5848987bf0d46c5555e06348c3549b53eb2d: Status 404 returned error can't find the container with id 1f5366190fe17fadfab59fa0c09b5848987bf0d46c5555e06348c3549b53eb2d Jul 06 21:07:55.468330 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.468308 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:07:55.617181 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:55.617159 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf"] Jul 06 21:07:55.618947 ip-10-0-142-142 kubenswrapper[2582]: W0706 21:07:55.618915 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bd7197_a4d2_4c4b_9cbc_83f4a2f2d958.slice/crio-e767aa69e64d357b847285b3365d78b6932037e215b10ee949d254d1cf5fbd17 WatchSource:0}: Error finding container e767aa69e64d357b847285b3365d78b6932037e215b10ee949d254d1cf5fbd17: Status 404 returned error can't find the container with id e767aa69e64d357b847285b3365d78b6932037e215b10ee949d254d1cf5fbd17 Jul 06 21:07:56.341434 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:56.341395 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" event={"ID":"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958","Type":"ContainerStarted","Data":"f68f54e1aac9820e6c2f52243f9edba448bfb59ac11ba191aa05b441c35088be"} Jul 06 21:07:56.342048 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:56.342017 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" event={"ID":"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958","Type":"ContainerStarted","Data":"3694047099c57075f5ad64f9ebd32d71a11e8e87d5b662fb84187f4955bbd19b"} Jul 06 21:07:56.342202 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:56.342185 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" event={"ID":"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958","Type":"ContainerStarted","Data":"e767aa69e64d357b847285b3365d78b6932037e215b10ee949d254d1cf5fbd17"} Jul 06 21:07:56.342328 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:56.342315 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:07:56.344867 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:56.344843 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:07:56.345011 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:56.344535 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 06 21:07:56.346081 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:56.346063 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" event={"ID":"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e","Type":"ContainerStarted","Data":"adf695c314b21cf79d92bf2692f5fabbd23d63accf5363f8c6e6700c90b9f3f3"} Jul 06 21:07:56.346172 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:56.346101 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" event={"ID":"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e","Type":"ContainerStarted","Data":"6cab28881f46706b2d426e10cb3bf54e194342c59fb4ac003837bd3378563fe3"} Jul 06 21:07:56.346172 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:56.346137 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" event={"ID":"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e","Type":"ContainerStarted","Data":"1f5366190fe17fadfab59fa0c09b5848987bf0d46c5555e06348c3549b53eb2d"} Jul 06 21:07:56.346279 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:56.346216 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:07:56.374825 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:56.374775 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" podStartSLOduration=1.374745761 podStartE2EDuration="1.374745761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:07:56.37154329 +0000 UTC m=+1943.457098483" watchObservedRunningTime="2026-07-06 21:07:56.374745761 +0000 UTC m=+1943.460300954" Jul 06 21:07:56.407660 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:56.407605 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" podStartSLOduration=2.407589545 podStartE2EDuration="2.407589545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:07:56.406496869 +0000 UTC m=+1943.492052062" watchObservedRunningTime="2026-07-06 21:07:56.407589545 +0000 UTC m=+1943.493144737" Jul 06 21:07:57.352541 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:57.352504 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:07:57.353044 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:57.352574 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 06 21:07:57.354083 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:57.354055 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Jul 06 21:07:57.793836 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:57.793715 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused" Jul 06 21:07:57.798184 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:57.798133 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" Jul 06 21:07:58.357000 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:58.356933 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Jul 06 21:07:58.357388 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:58.357010 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 06 21:07:58.801621 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:58.801526 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused" Jul 06 21:07:58.805901 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:58.805876 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused" Jul 06 21:07:59.293674 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.293642 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 21:07:59.338936 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.338855 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdcfb\" (UniqueName: \"kubernetes.io/projected/6611d492-131b-4cb5-81e2-e67702aac1f2-kube-api-access-gdcfb\") pod \"6611d492-131b-4cb5-81e2-e67702aac1f2\" (UID: \"6611d492-131b-4cb5-81e2-e67702aac1f2\") " Jul 06 21:07:59.338936 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.338899 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6611d492-131b-4cb5-81e2-e67702aac1f2-proxy-tls\") pod \"6611d492-131b-4cb5-81e2-e67702aac1f2\" (UID: \"6611d492-131b-4cb5-81e2-e67702aac1f2\") " Jul 06 21:07:59.339146 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.338957 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6611d492-131b-4cb5-81e2-e67702aac1f2-kube-rbac-proxy-sar-config\") pod \"6611d492-131b-4cb5-81e2-e67702aac1f2\" (UID: \"6611d492-131b-4cb5-81e2-e67702aac1f2\") " Jul 06 21:07:59.340471 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.340430 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6611d492-131b-4cb5-81e2-e67702aac1f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6611d492-131b-4cb5-81e2-e67702aac1f2" (UID: "6611d492-131b-4cb5-81e2-e67702aac1f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:07:59.342003 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.341965 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6611d492-131b-4cb5-81e2-e67702aac1f2-kube-api-access-gdcfb" (OuterVolumeSpecName: "kube-api-access-gdcfb") pod "6611d492-131b-4cb5-81e2-e67702aac1f2" (UID: "6611d492-131b-4cb5-81e2-e67702aac1f2"). InnerVolumeSpecName "kube-api-access-gdcfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:07:59.343243 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.343217 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6611d492-131b-4cb5-81e2-e67702aac1f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6611d492-131b-4cb5-81e2-e67702aac1f2" (UID: "6611d492-131b-4cb5-81e2-e67702aac1f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:07:59.357613 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.357591 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 21:07:59.363496 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.363471 2582 generic.go:356] "Generic (PLEG): container finished" podID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerID="3d361b7695f739d4ac42643c2ed06aab46519d1e90756b4b494c6c6603ebb4ef" exitCode=0 Jul 06 21:07:59.363588 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.363511 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" Jul 06 21:07:59.363588 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.363558 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" event={"ID":"6611d492-131b-4cb5-81e2-e67702aac1f2","Type":"ContainerDied","Data":"3d361b7695f739d4ac42643c2ed06aab46519d1e90756b4b494c6c6603ebb4ef"} Jul 06 21:07:59.363692 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.363606 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt" event={"ID":"6611d492-131b-4cb5-81e2-e67702aac1f2","Type":"ContainerDied","Data":"ce585c6dfc0ca706639074356c0a9eb5ca8ca7b625d8a9538621ff175b712800"} Jul 06 21:07:59.363692 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.363632 2582 scope.go:117] "RemoveContainer" containerID="6cc3b399501e28eeada8f6c90eb1c0972cd9efd7e607530dbf78d133f7732bcd" Jul 06 21:07:59.365187 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.365154 2582 generic.go:356] "Generic (PLEG): container finished" podID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerID="6d87cf3a52332597f96a1bd6e76ca5a4b8b1854f7f0b4f3afde9762a5dd8db9a" exitCode=0 Jul 06 21:07:59.365269 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.365189 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" event={"ID":"fc66437a-9311-48cd-963f-ac035c17b4a3","Type":"ContainerDied","Data":"6d87cf3a52332597f96a1bd6e76ca5a4b8b1854f7f0b4f3afde9762a5dd8db9a"} Jul 06 21:07:59.365269 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.365222 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" Jul 06 21:07:59.365269 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.365265 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv" event={"ID":"fc66437a-9311-48cd-963f-ac035c17b4a3","Type":"ContainerDied","Data":"15a062168b3296d3c38daa68dfd4dc4325a79e37b76445da351676a94e816a39"} Jul 06 21:07:59.372557 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.372541 2582 scope.go:117] "RemoveContainer" containerID="3d361b7695f739d4ac42643c2ed06aab46519d1e90756b4b494c6c6603ebb4ef" Jul 06 21:07:59.379550 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.379534 2582 scope.go:117] "RemoveContainer" containerID="6cc3b399501e28eeada8f6c90eb1c0972cd9efd7e607530dbf78d133f7732bcd" Jul 06 21:07:59.379838 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:07:59.379751 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cc3b399501e28eeada8f6c90eb1c0972cd9efd7e607530dbf78d133f7732bcd\": container with ID starting with 6cc3b399501e28eeada8f6c90eb1c0972cd9efd7e607530dbf78d133f7732bcd not found: ID does not exist" containerID="6cc3b399501e28eeada8f6c90eb1c0972cd9efd7e607530dbf78d133f7732bcd" Jul 06 21:07:59.379897 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.379849 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cc3b399501e28eeada8f6c90eb1c0972cd9efd7e607530dbf78d133f7732bcd"} err="failed to get container status \"6cc3b399501e28eeada8f6c90eb1c0972cd9efd7e607530dbf78d133f7732bcd\": rpc error: code = NotFound desc = could not find container \"6cc3b399501e28eeada8f6c90eb1c0972cd9efd7e607530dbf78d133f7732bcd\": container with ID starting with 6cc3b399501e28eeada8f6c90eb1c0972cd9efd7e607530dbf78d133f7732bcd not found: ID does not exist" Jul 06 21:07:59.379897 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.379870 2582 scope.go:117] "RemoveContainer" containerID="3d361b7695f739d4ac42643c2ed06aab46519d1e90756b4b494c6c6603ebb4ef" Jul 06 21:07:59.380148 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:07:59.380098 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d361b7695f739d4ac42643c2ed06aab46519d1e90756b4b494c6c6603ebb4ef\": container with ID starting with 3d361b7695f739d4ac42643c2ed06aab46519d1e90756b4b494c6c6603ebb4ef not found: ID does not exist" containerID="3d361b7695f739d4ac42643c2ed06aab46519d1e90756b4b494c6c6603ebb4ef" Jul 06 21:07:59.380214 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.380155 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d361b7695f739d4ac42643c2ed06aab46519d1e90756b4b494c6c6603ebb4ef"} err="failed to get container status \"3d361b7695f739d4ac42643c2ed06aab46519d1e90756b4b494c6c6603ebb4ef\": rpc error: code = NotFound desc = could not find container \"3d361b7695f739d4ac42643c2ed06aab46519d1e90756b4b494c6c6603ebb4ef\": container with ID starting with 3d361b7695f739d4ac42643c2ed06aab46519d1e90756b4b494c6c6603ebb4ef not found: ID does not exist" Jul 06 21:07:59.380214 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.380176 2582 scope.go:117] "RemoveContainer" containerID="3fb74d3e299917e82fe3b78b7418cf0a420d26d52496318defad5ad32a609444" Jul 06 21:07:59.388212 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.388196 2582 scope.go:117] "RemoveContainer" containerID="6d87cf3a52332597f96a1bd6e76ca5a4b8b1854f7f0b4f3afde9762a5dd8db9a" Jul 06 21:07:59.396254 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.396234 2582 scope.go:117] "RemoveContainer" containerID="3fb74d3e299917e82fe3b78b7418cf0a420d26d52496318defad5ad32a609444" Jul 06 21:07:59.396506 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:07:59.396490 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb74d3e299917e82fe3b78b7418cf0a420d26d52496318defad5ad32a609444\": container with ID starting with 3fb74d3e299917e82fe3b78b7418cf0a420d26d52496318defad5ad32a609444 not found: ID does not exist" containerID="3fb74d3e299917e82fe3b78b7418cf0a420d26d52496318defad5ad32a609444" Jul 06 21:07:59.396594 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.396517 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb74d3e299917e82fe3b78b7418cf0a420d26d52496318defad5ad32a609444"} err="failed to get container status \"3fb74d3e299917e82fe3b78b7418cf0a420d26d52496318defad5ad32a609444\": rpc error: code = NotFound desc = could not find container \"3fb74d3e299917e82fe3b78b7418cf0a420d26d52496318defad5ad32a609444\": container with ID starting with 3fb74d3e299917e82fe3b78b7418cf0a420d26d52496318defad5ad32a609444 not found: ID does not exist" Jul 06 21:07:59.396594 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.396538 2582 scope.go:117] "RemoveContainer" containerID="6d87cf3a52332597f96a1bd6e76ca5a4b8b1854f7f0b4f3afde9762a5dd8db9a" Jul 06 21:07:59.396898 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:07:59.396880 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d87cf3a52332597f96a1bd6e76ca5a4b8b1854f7f0b4f3afde9762a5dd8db9a\": container with ID starting with 6d87cf3a52332597f96a1bd6e76ca5a4b8b1854f7f0b4f3afde9762a5dd8db9a not found: ID does not exist" containerID="6d87cf3a52332597f96a1bd6e76ca5a4b8b1854f7f0b4f3afde9762a5dd8db9a" Jul 06 21:07:59.396970 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.396903 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d87cf3a52332597f96a1bd6e76ca5a4b8b1854f7f0b4f3afde9762a5dd8db9a"} err="failed to get container status \"6d87cf3a52332597f96a1bd6e76ca5a4b8b1854f7f0b4f3afde9762a5dd8db9a\": rpc error: code = NotFound desc = could not find container \"6d87cf3a52332597f96a1bd6e76ca5a4b8b1854f7f0b4f3afde9762a5dd8db9a\": container with ID starting with 6d87cf3a52332597f96a1bd6e76ca5a4b8b1854f7f0b4f3afde9762a5dd8db9a not found: ID does not exist" Jul 06 21:07:59.402847 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.402827 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt"] Jul 06 21:07:59.409630 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.409610 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-92a61-predictor-f78d96bf8-l44xt"] Jul 06 21:07:59.440234 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.440206 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc66437a-9311-48cd-963f-ac035c17b4a3-proxy-tls\") pod \"fc66437a-9311-48cd-963f-ac035c17b4a3\" (UID: \"fc66437a-9311-48cd-963f-ac035c17b4a3\") " Jul 06 21:07:59.440345 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.440278 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc66437a-9311-48cd-963f-ac035c17b4a3-kube-rbac-proxy-sar-config\") pod \"fc66437a-9311-48cd-963f-ac035c17b4a3\" (UID: \"fc66437a-9311-48cd-963f-ac035c17b4a3\") " Jul 06 21:07:59.440402 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.440356 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfl92\" (UniqueName: \"kubernetes.io/projected/fc66437a-9311-48cd-963f-ac035c17b4a3-kube-api-access-xfl92\") pod \"fc66437a-9311-48cd-963f-ac035c17b4a3\" (UID: \"fc66437a-9311-48cd-963f-ac035c17b4a3\") " Jul 06 21:07:59.440584 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.440565 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdcfb\" (UniqueName: \"kubernetes.io/projected/6611d492-131b-4cb5-81e2-e67702aac1f2-kube-api-access-gdcfb\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:07:59.440744 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.440729 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6611d492-131b-4cb5-81e2-e67702aac1f2-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:07:59.440887 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.440869 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6611d492-131b-4cb5-81e2-e67702aac1f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:07:59.440887 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.440657 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc66437a-9311-48cd-963f-ac035c17b4a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc66437a-9311-48cd-963f-ac035c17b4a3" (UID: "fc66437a-9311-48cd-963f-ac035c17b4a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:07:59.442532 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.442511 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc66437a-9311-48cd-963f-ac035c17b4a3-kube-api-access-xfl92" (OuterVolumeSpecName: "kube-api-access-xfl92") pod "fc66437a-9311-48cd-963f-ac035c17b4a3" (UID: "fc66437a-9311-48cd-963f-ac035c17b4a3"). InnerVolumeSpecName "kube-api-access-xfl92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:07:59.442753 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.442738 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc66437a-9311-48cd-963f-ac035c17b4a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc66437a-9311-48cd-963f-ac035c17b4a3" (UID: "fc66437a-9311-48cd-963f-ac035c17b4a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:07:59.537956 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.537926 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" path="/var/lib/kubelet/pods/6611d492-131b-4cb5-81e2-e67702aac1f2/volumes" Jul 06 21:07:59.542244 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.542213 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc66437a-9311-48cd-963f-ac035c17b4a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:07:59.542244 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.542246 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfl92\" (UniqueName: \"kubernetes.io/projected/fc66437a-9311-48cd-963f-ac035c17b4a3-kube-api-access-xfl92\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:07:59.542447 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.542262 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc66437a-9311-48cd-963f-ac035c17b4a3-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:07:59.688129 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.688099 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv"] Jul 06 21:07:59.694804 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:07:59.694780 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-92a61-predictor-5b8cf99c8c-75dxv"] Jul 06 21:08:01.538993 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:01.538956 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" path="/var/lib/kubelet/pods/fc66437a-9311-48cd-963f-ac035c17b4a3/volumes" Jul 06 21:08:02.201140 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:02.201101 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused" Jul 06 21:08:03.207889 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:03.207849 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused" Jul 06 21:08:03.361145 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:03.361111 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:08:03.361775 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:03.361738 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:08:03.361775 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:03.361743 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 06 21:08:03.362211 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:03.362185 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Jul 06 21:08:12.202025 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:12.201989 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:08:13.208945 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:13.208908 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:08:13.362574 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:13.362541 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Jul 06 21:08:13.362737 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:13.362541 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 06 21:08:23.361769 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:23.361714 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 06 21:08:23.362263 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:23.362237 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Jul 06 21:08:33.362237 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:33.362160 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused" Jul 06 21:08:33.362599 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:33.362164 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused" Jul 06 21:08:35.337780 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.334917 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw"] Jul 06 21:08:35.337780 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.335299 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kserve-container" containerID="cri-o://7b758bbeca9ff940fcf2798cd30da526da54a7d5ecf2a8902050781ddc87e1ca" gracePeriod=30 Jul 06 21:08:35.337780 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.335520 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kube-rbac-proxy" containerID="cri-o://88dfc3232725fe4912434fb592c3a648ccb8e6641a4153521029da3759d18c73" gracePeriod=30 Jul 06 21:08:35.418373 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.418339 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn"] Jul 06 21:08:35.418704 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.418693 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kserve-container" Jul 06 21:08:35.418750 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.418706 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kserve-container" Jul 06 21:08:35.418750 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.418718 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kube-rbac-proxy" Jul 06 21:08:35.418750 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.418724 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kube-rbac-proxy" Jul 06 21:08:35.418750 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.418735 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kserve-container" Jul 06 21:08:35.418750 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.418741 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kserve-container" Jul 06 21:08:35.418750 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.418747 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kube-rbac-proxy" Jul 06 21:08:35.418750 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.418751 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kube-rbac-proxy" Jul 06 21:08:35.418961 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.418813 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kserve-container" Jul 06 21:08:35.418961 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.418821 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6611d492-131b-4cb5-81e2-e67702aac1f2" containerName="kube-rbac-proxy" Jul 06 21:08:35.418961 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.418827 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kserve-container" Jul 06 21:08:35.418961 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.418832 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc66437a-9311-48cd-963f-ac035c17b4a3" containerName="kube-rbac-proxy" Jul 06 21:08:35.422997 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.422982 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:35.427369 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.427353 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d7086-predictor-serving-cert\"" Jul 06 21:08:35.428398 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.428380 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d7086-kube-rbac-proxy-sar-config\"" Jul 06 21:08:35.435556 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.435529 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn"] Jul 06 21:08:35.473410 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.473386 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m"] Jul 06 21:08:35.473595 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.473571 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kserve-container" containerID="cri-o://762e86aa0c0adde717094bca646286124858e1f9e5b1b8853ce40b887049914d" gracePeriod=30 Jul 06 21:08:35.473714 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.473599 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kube-rbac-proxy" containerID="cri-o://1faa1f69cd65da5737e8553069b40a75b0fdad7ba50d8460e0efb85b908d67cb" gracePeriod=30 Jul 06 21:08:35.547898 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.547857 2582 generic.go:356] "Generic (PLEG): container finished" podID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerID="88dfc3232725fe4912434fb592c3a648ccb8e6641a4153521029da3759d18c73" exitCode=2 Jul 06 21:08:35.548052 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.547913 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" event={"ID":"46b44663-269b-4ead-bbd4-09f1909c4e9b","Type":"ContainerDied","Data":"88dfc3232725fe4912434fb592c3a648ccb8e6641a4153521029da3759d18c73"} Jul 06 21:08:35.551241 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.551223 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j9sk\" (UniqueName: \"kubernetes.io/projected/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-kube-api-access-9j9sk\") pod \"success-200-isvc-d7086-predictor-6848586898-5cvmn\" (UID: \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\") " pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:35.551294 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.551252 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d7086-predictor-6848586898-5cvmn\" (UID: \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\") " pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:35.551333 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.551293 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-proxy-tls\") pod \"success-200-isvc-d7086-predictor-6848586898-5cvmn\" (UID: \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\") " pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:35.577984 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.577955 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2"] Jul 06 21:08:35.581590 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.581567 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:35.587507 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.587491 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d7086-predictor-serving-cert\"" Jul 06 21:08:35.587850 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.587800 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d7086-kube-rbac-proxy-sar-config\"" Jul 06 21:08:35.603614 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.603586 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2"] Jul 06 21:08:35.652340 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.652303 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j9sk\" (UniqueName: \"kubernetes.io/projected/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-kube-api-access-9j9sk\") pod \"success-200-isvc-d7086-predictor-6848586898-5cvmn\" (UID: \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\") " pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:35.652340 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.652342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d7086-predictor-6848586898-5cvmn\" (UID: \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\") " pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:35.652614 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.652589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-proxy-tls\") pod \"success-200-isvc-d7086-predictor-6848586898-5cvmn\" (UID: \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\") " pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:35.652693 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:08:35.652676 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d7086-predictor-serving-cert: secret "success-200-isvc-d7086-predictor-serving-cert" not found Jul 06 21:08:35.652779 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:08:35.652743 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-proxy-tls podName:a34b63b4-ac86-40cf-9b74-fe0cb52f9b75 nodeName:}" failed. No retries permitted until 2026-07-06 21:08:36.152724143 +0000 UTC m=+1983.238279319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-proxy-tls") pod "success-200-isvc-d7086-predictor-6848586898-5cvmn" (UID: "a34b63b4-ac86-40cf-9b74-fe0cb52f9b75") : secret "success-200-isvc-d7086-predictor-serving-cert" not found Jul 06 21:08:35.653041 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.653019 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d7086-predictor-6848586898-5cvmn\" (UID: \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\") " pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:35.667599 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.667571 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j9sk\" (UniqueName: \"kubernetes.io/projected/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-kube-api-access-9j9sk\") pod \"success-200-isvc-d7086-predictor-6848586898-5cvmn\" (UID: \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\") " pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:35.753319 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.753290 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed5be343-7cc7-4eca-8436-1bf8c5dda709-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d7086-predictor-84cb8df759-4cmm2\" (UID: \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\") " pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:35.753442 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.753350 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knb6n\" (UniqueName: \"kubernetes.io/projected/ed5be343-7cc7-4eca-8436-1bf8c5dda709-kube-api-access-knb6n\") pod \"error-404-isvc-d7086-predictor-84cb8df759-4cmm2\" (UID: \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\") " pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:35.753442 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.753411 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed5be343-7cc7-4eca-8436-1bf8c5dda709-proxy-tls\") pod \"error-404-isvc-d7086-predictor-84cb8df759-4cmm2\" (UID: \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\") " pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:35.854575 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.854549 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knb6n\" (UniqueName: \"kubernetes.io/projected/ed5be343-7cc7-4eca-8436-1bf8c5dda709-kube-api-access-knb6n\") pod \"error-404-isvc-d7086-predictor-84cb8df759-4cmm2\" (UID: \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\") " pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:35.854695 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.854584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed5be343-7cc7-4eca-8436-1bf8c5dda709-proxy-tls\") pod \"error-404-isvc-d7086-predictor-84cb8df759-4cmm2\" (UID: \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\") " pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:35.854695 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.854648 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed5be343-7cc7-4eca-8436-1bf8c5dda709-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d7086-predictor-84cb8df759-4cmm2\" (UID: \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\") " pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:35.854838 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:08:35.854786 2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d7086-predictor-serving-cert: secret "error-404-isvc-d7086-predictor-serving-cert" not found Jul 06 21:08:35.854887 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:08:35.854846 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed5be343-7cc7-4eca-8436-1bf8c5dda709-proxy-tls podName:ed5be343-7cc7-4eca-8436-1bf8c5dda709 nodeName:}" failed. No retries permitted until 2026-07-06 21:08:36.354832704 +0000 UTC m=+1983.440387875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ed5be343-7cc7-4eca-8436-1bf8c5dda709-proxy-tls") pod "error-404-isvc-d7086-predictor-84cb8df759-4cmm2" (UID: "ed5be343-7cc7-4eca-8436-1bf8c5dda709") : secret "error-404-isvc-d7086-predictor-serving-cert" not found Jul 06 21:08:35.855257 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.855240 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed5be343-7cc7-4eca-8436-1bf8c5dda709-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d7086-predictor-84cb8df759-4cmm2\" (UID: \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\") " pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:35.867513 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:35.867481 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knb6n\" (UniqueName: \"kubernetes.io/projected/ed5be343-7cc7-4eca-8436-1bf8c5dda709-kube-api-access-knb6n\") pod \"error-404-isvc-d7086-predictor-84cb8df759-4cmm2\" (UID: \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\") " pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:36.157988 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:36.157899 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-proxy-tls\") pod \"success-200-isvc-d7086-predictor-6848586898-5cvmn\" (UID: \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\") " pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:36.160120 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:36.160089 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-proxy-tls\") pod \"success-200-isvc-d7086-predictor-6848586898-5cvmn\" (UID: \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\") " pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:36.333012 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:36.332978 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:36.359400 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:36.359371 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed5be343-7cc7-4eca-8436-1bf8c5dda709-proxy-tls\") pod \"error-404-isvc-d7086-predictor-84cb8df759-4cmm2\" (UID: \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\") " pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:36.361807 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:36.361780 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed5be343-7cc7-4eca-8436-1bf8c5dda709-proxy-tls\") pod \"error-404-isvc-d7086-predictor-84cb8df759-4cmm2\" (UID: \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\") " pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:36.486996 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:36.486967 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn"] Jul 06 21:08:36.490972 ip-10-0-142-142 kubenswrapper[2582]: W0706 21:08:36.490939 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34b63b4_ac86_40cf_9b74_fe0cb52f9b75.slice/crio-b86feacc819b490e78995ce48c09e07eea665a21fae255fc93cf2d666f2dffff WatchSource:0}: Error finding container b86feacc819b490e78995ce48c09e07eea665a21fae255fc93cf2d666f2dffff: Status 404 returned error can't find the container with id b86feacc819b490e78995ce48c09e07eea665a21fae255fc93cf2d666f2dffff Jul 06 21:08:36.491688 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:36.491664 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:36.557131 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:36.557103 2582 generic.go:356] "Generic (PLEG): container finished" podID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerID="1faa1f69cd65da5737e8553069b40a75b0fdad7ba50d8460e0efb85b908d67cb" exitCode=2 Jul 06 21:08:36.557244 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:36.557179 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" event={"ID":"fae36521-e6ea-446b-892c-51cc9ec16d81","Type":"ContainerDied","Data":"1faa1f69cd65da5737e8553069b40a75b0fdad7ba50d8460e0efb85b908d67cb"} Jul 06 21:08:36.559008 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:36.558985 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" event={"ID":"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75","Type":"ContainerStarted","Data":"b86feacc819b490e78995ce48c09e07eea665a21fae255fc93cf2d666f2dffff"} Jul 06 21:08:36.643265 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:36.643209 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2"] Jul 06 21:08:36.647715 ip-10-0-142-142 kubenswrapper[2582]: W0706 21:08:36.647678 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5be343_7cc7_4eca_8436_1bf8c5dda709.slice/crio-a64bb69e98037ecb5d362cd71779aaa402f1007822819b909c6921990e701f9a WatchSource:0}: Error finding container a64bb69e98037ecb5d362cd71779aaa402f1007822819b909c6921990e701f9a: Status 404 returned error can't find the container with id a64bb69e98037ecb5d362cd71779aaa402f1007822819b909c6921990e701f9a Jul 06 21:08:37.196588 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:37.196543 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused" Jul 06 21:08:37.565369 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:37.565271 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" event={"ID":"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75","Type":"ContainerStarted","Data":"717ac4962813bed09ad7cc07cbf5aa3eaf704a6dded45d87e146bbed0d19aa6b"} Jul 06 21:08:37.565809 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:37.565448 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:37.565809 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:37.565496 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:37.565809 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:37.565531 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" event={"ID":"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75","Type":"ContainerStarted","Data":"7ce5f1b6a2ed571a361227480fe620e83b220d4f49edba45790139491655c16b"} Jul 06 21:08:37.566803 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:37.566777 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused" Jul 06 21:08:37.567090 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:37.567072 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" event={"ID":"ed5be343-7cc7-4eca-8436-1bf8c5dda709","Type":"ContainerStarted","Data":"f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92"} Jul 06 21:08:37.567150 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:37.567110 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" event={"ID":"ed5be343-7cc7-4eca-8436-1bf8c5dda709","Type":"ContainerStarted","Data":"6d5d5f67bf49bc09f890f0eb54f872bb988515bfa7534f360cdda53d4b03d85f"} Jul 06 21:08:37.567150 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:37.567129 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" event={"ID":"ed5be343-7cc7-4eca-8436-1bf8c5dda709","Type":"ContainerStarted","Data":"a64bb69e98037ecb5d362cd71779aaa402f1007822819b909c6921990e701f9a"} Jul 06 21:08:37.567227 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:37.567200 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:37.591218 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:37.591176 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" podStartSLOduration=2.5911635349999997 podStartE2EDuration="2.591163535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:37.589880993 +0000 UTC m=+1984.675436191" watchObservedRunningTime="2026-07-06 21:08:37.591163535 +0000 UTC m=+1984.676718798" Jul 06 21:08:37.620213 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:37.620158 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" podStartSLOduration=2.62014287 podStartE2EDuration="2.62014287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:37.618001904 +0000 UTC m=+1984.703557122" watchObservedRunningTime="2026-07-06 21:08:37.62014287 +0000 UTC m=+1984.705698065" Jul 06 21:08:38.202975 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.202931 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused" Jul 06 21:08:38.518876 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.518852 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:08:38.573962 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.573927 2582 generic.go:356] "Generic (PLEG): container finished" podID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerID="762e86aa0c0adde717094bca646286124858e1f9e5b1b8853ce40b887049914d" exitCode=0 Jul 06 21:08:38.574364 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.573979 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" Jul 06 21:08:38.574364 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.574004 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" event={"ID":"fae36521-e6ea-446b-892c-51cc9ec16d81","Type":"ContainerDied","Data":"762e86aa0c0adde717094bca646286124858e1f9e5b1b8853ce40b887049914d"} Jul 06 21:08:38.574364 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.574075 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m" event={"ID":"fae36521-e6ea-446b-892c-51cc9ec16d81","Type":"ContainerDied","Data":"3f5f016225c7eaadd97d82d536ef4269c3889dd864069ee317de0e736b4ff4c8"} Jul 06 21:08:38.574364 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.574106 2582 scope.go:117] "RemoveContainer" containerID="1faa1f69cd65da5737e8553069b40a75b0fdad7ba50d8460e0efb85b908d67cb" Jul 06 21:08:38.574591 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.574545 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:38.574658 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.574637 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused" Jul 06 21:08:38.575700 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.575680 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Jul 06 21:08:38.582499 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.582484 2582 scope.go:117] "RemoveContainer" containerID="762e86aa0c0adde717094bca646286124858e1f9e5b1b8853ce40b887049914d" Jul 06 21:08:38.590048 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.590029 2582 scope.go:117] "RemoveContainer" containerID="1faa1f69cd65da5737e8553069b40a75b0fdad7ba50d8460e0efb85b908d67cb" Jul 06 21:08:38.590325 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:08:38.590305 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1faa1f69cd65da5737e8553069b40a75b0fdad7ba50d8460e0efb85b908d67cb\": container with ID starting with 1faa1f69cd65da5737e8553069b40a75b0fdad7ba50d8460e0efb85b908d67cb not found: ID does not exist" containerID="1faa1f69cd65da5737e8553069b40a75b0fdad7ba50d8460e0efb85b908d67cb" Jul 06 21:08:38.590375 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.590333 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1faa1f69cd65da5737e8553069b40a75b0fdad7ba50d8460e0efb85b908d67cb"} err="failed to get container status \"1faa1f69cd65da5737e8553069b40a75b0fdad7ba50d8460e0efb85b908d67cb\": rpc error: code = NotFound desc = could not find container \"1faa1f69cd65da5737e8553069b40a75b0fdad7ba50d8460e0efb85b908d67cb\": container with ID starting with 1faa1f69cd65da5737e8553069b40a75b0fdad7ba50d8460e0efb85b908d67cb not found: ID does not exist" Jul 06 21:08:38.590375 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.590351 2582 scope.go:117] "RemoveContainer" containerID="762e86aa0c0adde717094bca646286124858e1f9e5b1b8853ce40b887049914d" Jul 06 21:08:38.590580 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:08:38.590564 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762e86aa0c0adde717094bca646286124858e1f9e5b1b8853ce40b887049914d\": container with ID starting with 762e86aa0c0adde717094bca646286124858e1f9e5b1b8853ce40b887049914d not found: ID does not exist" containerID="762e86aa0c0adde717094bca646286124858e1f9e5b1b8853ce40b887049914d" Jul 06 21:08:38.590616 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.590588 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762e86aa0c0adde717094bca646286124858e1f9e5b1b8853ce40b887049914d"} err="failed to get container status \"762e86aa0c0adde717094bca646286124858e1f9e5b1b8853ce40b887049914d\": rpc error: code = NotFound desc = could not find container \"762e86aa0c0adde717094bca646286124858e1f9e5b1b8853ce40b887049914d\": container with ID starting with 762e86aa0c0adde717094bca646286124858e1f9e5b1b8853ce40b887049914d not found: ID does not exist" Jul 06 21:08:38.680595 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.680572 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ndn4\" (UniqueName: \"kubernetes.io/projected/fae36521-e6ea-446b-892c-51cc9ec16d81-kube-api-access-4ndn4\") pod \"fae36521-e6ea-446b-892c-51cc9ec16d81\" (UID: \"fae36521-e6ea-446b-892c-51cc9ec16d81\") " Jul 06 21:08:38.680687 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.680600 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fae36521-e6ea-446b-892c-51cc9ec16d81-proxy-tls\") pod \"fae36521-e6ea-446b-892c-51cc9ec16d81\" (UID: \"fae36521-e6ea-446b-892c-51cc9ec16d81\") " Jul 06 21:08:38.680687 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.680678 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fae36521-e6ea-446b-892c-51cc9ec16d81-kube-rbac-proxy-sar-config\") pod \"fae36521-e6ea-446b-892c-51cc9ec16d81\" (UID: \"fae36521-e6ea-446b-892c-51cc9ec16d81\") " Jul 06 21:08:38.681295 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.681127 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae36521-e6ea-446b-892c-51cc9ec16d81-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fae36521-e6ea-446b-892c-51cc9ec16d81" (UID: "fae36521-e6ea-446b-892c-51cc9ec16d81"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:08:38.682691 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.682669 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae36521-e6ea-446b-892c-51cc9ec16d81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fae36521-e6ea-446b-892c-51cc9ec16d81" (UID: "fae36521-e6ea-446b-892c-51cc9ec16d81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:08:38.682832 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.682789 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae36521-e6ea-446b-892c-51cc9ec16d81-kube-api-access-4ndn4" (OuterVolumeSpecName: "kube-api-access-4ndn4") pod "fae36521-e6ea-446b-892c-51cc9ec16d81" (UID: "fae36521-e6ea-446b-892c-51cc9ec16d81"). InnerVolumeSpecName "kube-api-access-4ndn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:08:38.788681 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.788648 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fae36521-e6ea-446b-892c-51cc9ec16d81-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:08:38.788842 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.788692 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ndn4\" (UniqueName: \"kubernetes.io/projected/fae36521-e6ea-446b-892c-51cc9ec16d81-kube-api-access-4ndn4\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:08:38.788842 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.788708 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fae36521-e6ea-446b-892c-51cc9ec16d81-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:08:38.880476 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.880454 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:08:38.889732 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.889714 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk5p2\" (UniqueName: \"kubernetes.io/projected/46b44663-269b-4ead-bbd4-09f1909c4e9b-kube-api-access-wk5p2\") pod \"46b44663-269b-4ead-bbd4-09f1909c4e9b\" (UID: \"46b44663-269b-4ead-bbd4-09f1909c4e9b\") " Jul 06 21:08:38.889831 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.889749 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46b44663-269b-4ead-bbd4-09f1909c4e9b-proxy-tls\") pod \"46b44663-269b-4ead-bbd4-09f1909c4e9b\" (UID: \"46b44663-269b-4ead-bbd4-09f1909c4e9b\") " Jul 06 21:08:38.889831 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.889808 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46b44663-269b-4ead-bbd4-09f1909c4e9b-kube-rbac-proxy-sar-config\") pod \"46b44663-269b-4ead-bbd4-09f1909c4e9b\" (UID: \"46b44663-269b-4ead-bbd4-09f1909c4e9b\") " Jul 06 21:08:38.890211 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.890181 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b44663-269b-4ead-bbd4-09f1909c4e9b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "46b44663-269b-4ead-bbd4-09f1909c4e9b" (UID: "46b44663-269b-4ead-bbd4-09f1909c4e9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:08:38.891569 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.891546 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b44663-269b-4ead-bbd4-09f1909c4e9b-kube-api-access-wk5p2" (OuterVolumeSpecName: "kube-api-access-wk5p2") pod "46b44663-269b-4ead-bbd4-09f1909c4e9b" (UID: "46b44663-269b-4ead-bbd4-09f1909c4e9b"). InnerVolumeSpecName "kube-api-access-wk5p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:08:38.891648 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.891600 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b44663-269b-4ead-bbd4-09f1909c4e9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "46b44663-269b-4ead-bbd4-09f1909c4e9b" (UID: "46b44663-269b-4ead-bbd4-09f1909c4e9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:08:38.919414 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.919388 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m"] Jul 06 21:08:38.924528 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.924507 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-999e1-predictor-57944785f4-q888m"] Jul 06 21:08:38.990460 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.990435 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk5p2\" (UniqueName: \"kubernetes.io/projected/46b44663-269b-4ead-bbd4-09f1909c4e9b-kube-api-access-wk5p2\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:08:38.990460 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.990457 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46b44663-269b-4ead-bbd4-09f1909c4e9b-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:08:38.990625 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:38.990472 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46b44663-269b-4ead-bbd4-09f1909c4e9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:08:39.539120 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.539081 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" path="/var/lib/kubelet/pods/fae36521-e6ea-446b-892c-51cc9ec16d81/volumes" Jul 06 21:08:39.580219 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.580189 2582 generic.go:356] "Generic (PLEG): container finished" podID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerID="7b758bbeca9ff940fcf2798cd30da526da54a7d5ecf2a8902050781ddc87e1ca" exitCode=0 Jul 06 21:08:39.580544 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.580242 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" Jul 06 21:08:39.580544 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.580276 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" event={"ID":"46b44663-269b-4ead-bbd4-09f1909c4e9b","Type":"ContainerDied","Data":"7b758bbeca9ff940fcf2798cd30da526da54a7d5ecf2a8902050781ddc87e1ca"} Jul 06 21:08:39.580544 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.580334 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw" event={"ID":"46b44663-269b-4ead-bbd4-09f1909c4e9b","Type":"ContainerDied","Data":"872bd080b8be5eb42a5493154747358081e45d6e7f269d813d7bc3c48b43fc95"} Jul 06 21:08:39.580544 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.580365 2582 scope.go:117] "RemoveContainer" containerID="88dfc3232725fe4912434fb592c3a648ccb8e6641a4153521029da3759d18c73" Jul 06 21:08:39.581042 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.581021 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Jul 06 21:08:39.588583 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.588556 2582 scope.go:117] "RemoveContainer" containerID="7b758bbeca9ff940fcf2798cd30da526da54a7d5ecf2a8902050781ddc87e1ca" Jul 06 21:08:39.596136 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.596120 2582 scope.go:117] "RemoveContainer" containerID="88dfc3232725fe4912434fb592c3a648ccb8e6641a4153521029da3759d18c73" Jul 06 21:08:39.596390 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:08:39.596375 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88dfc3232725fe4912434fb592c3a648ccb8e6641a4153521029da3759d18c73\": container with ID starting with 88dfc3232725fe4912434fb592c3a648ccb8e6641a4153521029da3759d18c73 not found: ID does not exist" containerID="88dfc3232725fe4912434fb592c3a648ccb8e6641a4153521029da3759d18c73" Jul 06 21:08:39.596438 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.596396 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88dfc3232725fe4912434fb592c3a648ccb8e6641a4153521029da3759d18c73"} err="failed to get container status \"88dfc3232725fe4912434fb592c3a648ccb8e6641a4153521029da3759d18c73\": rpc error: code = NotFound desc = could not find container \"88dfc3232725fe4912434fb592c3a648ccb8e6641a4153521029da3759d18c73\": container with ID starting with 88dfc3232725fe4912434fb592c3a648ccb8e6641a4153521029da3759d18c73 not found: ID does not exist" Jul 06 21:08:39.596438 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.596410 2582 scope.go:117] "RemoveContainer" containerID="7b758bbeca9ff940fcf2798cd30da526da54a7d5ecf2a8902050781ddc87e1ca" Jul 06 21:08:39.596599 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:08:39.596585 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b758bbeca9ff940fcf2798cd30da526da54a7d5ecf2a8902050781ddc87e1ca\": container with ID starting with 7b758bbeca9ff940fcf2798cd30da526da54a7d5ecf2a8902050781ddc87e1ca not found: ID does not exist" containerID="7b758bbeca9ff940fcf2798cd30da526da54a7d5ecf2a8902050781ddc87e1ca" Jul 06 21:08:39.596633 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.596601 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b758bbeca9ff940fcf2798cd30da526da54a7d5ecf2a8902050781ddc87e1ca"} err="failed to get container status \"7b758bbeca9ff940fcf2798cd30da526da54a7d5ecf2a8902050781ddc87e1ca\": rpc error: code = NotFound desc = could not find container \"7b758bbeca9ff940fcf2798cd30da526da54a7d5ecf2a8902050781ddc87e1ca\": container with ID starting with 7b758bbeca9ff940fcf2798cd30da526da54a7d5ecf2a8902050781ddc87e1ca not found: ID does not exist" Jul 06 21:08:39.614692 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.614670 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw"] Jul 06 21:08:39.621396 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:39.621376 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-999e1-predictor-fd6f8745f-njpnw"] Jul 06 21:08:41.538413 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:41.538369 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" path="/var/lib/kubelet/pods/46b44663-269b-4ead-bbd4-09f1909c4e9b/volumes" Jul 06 21:08:43.363440 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:43.363412 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:08:43.363835 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:43.363672 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:08:43.579228 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:43.579200 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:08:43.579630 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:43.579605 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused" Jul 06 21:08:44.585146 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:44.585118 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:08:44.585526 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:44.585500 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Jul 06 21:08:53.579813 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:53.579751 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused" Jul 06 21:08:54.586522 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:08:54.586481 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Jul 06 21:09:03.580280 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:09:03.580243 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused" Jul 06 21:09:04.586443 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:09:04.586409 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Jul 06 21:09:13.579781 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:09:13.579730 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused" Jul 06 21:09:14.586399 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:09:14.586359 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused" Jul 06 21:09:23.581252 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:09:23.581219 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:09:24.587578 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:09:24.587551 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:10:33.506909 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:10:33.506882 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 21:10:33.511958 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:10:33.511937 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 21:15:33.533861 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:15:33.533831 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 21:15:33.541115 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:15:33.541093 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 21:17:50.357165 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:50.357121 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn"] Jul 06 21:17:50.357755 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:50.357414 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kserve-container" containerID="cri-o://717ac4962813bed09ad7cc07cbf5aa3eaf704a6dded45d87e146bbed0d19aa6b" gracePeriod=30 Jul 06 21:17:50.357755 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:50.357437 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kube-rbac-proxy" containerID="cri-o://7ce5f1b6a2ed571a361227480fe620e83b220d4f49edba45790139491655c16b" gracePeriod=30 Jul 06 21:17:50.410579 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:50.410544 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2"] Jul 06 21:17:50.410853 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:50.410827 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kserve-container" containerID="cri-o://6d5d5f67bf49bc09f890f0eb54f872bb988515bfa7534f360cdda53d4b03d85f" gracePeriod=30 Jul 06 21:17:50.410939 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:50.410864 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kube-rbac-proxy" containerID="cri-o://f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92" gracePeriod=30 Jul 06 21:17:50.457573 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:17:50.457537 2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5be343_7cc7_4eca_8436_1bf8c5dda709.slice/crio-conmon-f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92.scope\": RecentStats: unable to find data in memory cache]" Jul 06 21:17:51.033457 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:51.033422 2582 generic.go:356] "Generic (PLEG): container finished" podID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerID="7ce5f1b6a2ed571a361227480fe620e83b220d4f49edba45790139491655c16b" exitCode=2 Jul 06 21:17:51.033646 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:51.033498 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" event={"ID":"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75","Type":"ContainerDied","Data":"7ce5f1b6a2ed571a361227480fe620e83b220d4f49edba45790139491655c16b"} Jul 06 21:17:51.035127 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:51.035107 2582 generic.go:356] "Generic (PLEG): container finished" podID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerID="f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92" exitCode=2 Jul 06 21:17:51.035238 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:51.035137 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" event={"ID":"ed5be343-7cc7-4eca-8436-1bf8c5dda709","Type":"ContainerDied","Data":"f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92"} Jul 06 21:17:53.637282 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.637015 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:17:53.748535 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.748452 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-proxy-tls\") pod \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\" (UID: \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\") " Jul 06 21:17:53.748535 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.748524 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j9sk\" (UniqueName: \"kubernetes.io/projected/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-kube-api-access-9j9sk\") pod \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\" (UID: \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\") " Jul 06 21:17:53.748774 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.748677 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-kube-rbac-proxy-sar-config\") pod \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\" (UID: \"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75\") " Jul 06 21:17:53.749047 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.749021 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" (UID: "a34b63b4-ac86-40cf-9b74-fe0cb52f9b75"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:17:53.750514 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.750490 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" (UID: "a34b63b4-ac86-40cf-9b74-fe0cb52f9b75"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:17:53.750657 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.750636 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-kube-api-access-9j9sk" (OuterVolumeSpecName: "kube-api-access-9j9sk") pod "a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" (UID: "a34b63b4-ac86-40cf-9b74-fe0cb52f9b75"). InnerVolumeSpecName "kube-api-access-9j9sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:17:53.759202 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.759184 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:17:53.849495 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.849457 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed5be343-7cc7-4eca-8436-1bf8c5dda709-kube-rbac-proxy-sar-config\") pod \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\" (UID: \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\") " Jul 06 21:17:53.849690 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.849567 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knb6n\" (UniqueName: \"kubernetes.io/projected/ed5be343-7cc7-4eca-8436-1bf8c5dda709-kube-api-access-knb6n\") pod \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\" (UID: \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\") " Jul 06 21:17:53.849690 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.849594 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed5be343-7cc7-4eca-8436-1bf8c5dda709-proxy-tls\") pod \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\" (UID: \"ed5be343-7cc7-4eca-8436-1bf8c5dda709\") " Jul 06 21:17:53.849863 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.849716 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:17:53.849863 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.849732 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j9sk\" (UniqueName: \"kubernetes.io/projected/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-kube-api-access-9j9sk\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:17:53.849863 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.849748 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:17:53.849992 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.849899 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5be343-7cc7-4eca-8436-1bf8c5dda709-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed5be343-7cc7-4eca-8436-1bf8c5dda709" (UID: "ed5be343-7cc7-4eca-8436-1bf8c5dda709"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:17:53.851654 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.851631 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5be343-7cc7-4eca-8436-1bf8c5dda709-kube-api-access-knb6n" (OuterVolumeSpecName: "kube-api-access-knb6n") pod "ed5be343-7cc7-4eca-8436-1bf8c5dda709" (UID: "ed5be343-7cc7-4eca-8436-1bf8c5dda709"). InnerVolumeSpecName "kube-api-access-knb6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:17:53.851726 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.851639 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5be343-7cc7-4eca-8436-1bf8c5dda709-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed5be343-7cc7-4eca-8436-1bf8c5dda709" (UID: "ed5be343-7cc7-4eca-8436-1bf8c5dda709"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:17:53.950706 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.950665 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knb6n\" (UniqueName: \"kubernetes.io/projected/ed5be343-7cc7-4eca-8436-1bf8c5dda709-kube-api-access-knb6n\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:17:53.950706 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.950700 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed5be343-7cc7-4eca-8436-1bf8c5dda709-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:17:53.950706 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:53.950710 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed5be343-7cc7-4eca-8436-1bf8c5dda709-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:17:54.051984 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.051895 2582 generic.go:356] "Generic (PLEG): container finished" podID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerID="6d5d5f67bf49bc09f890f0eb54f872bb988515bfa7534f360cdda53d4b03d85f" exitCode=0 Jul 06 21:17:54.051984 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.051961 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" Jul 06 21:17:54.052210 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.051974 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" event={"ID":"ed5be343-7cc7-4eca-8436-1bf8c5dda709","Type":"ContainerDied","Data":"6d5d5f67bf49bc09f890f0eb54f872bb988515bfa7534f360cdda53d4b03d85f"} Jul 06 21:17:54.052210 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.052070 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2" event={"ID":"ed5be343-7cc7-4eca-8436-1bf8c5dda709","Type":"ContainerDied","Data":"a64bb69e98037ecb5d362cd71779aaa402f1007822819b909c6921990e701f9a"} Jul 06 21:17:54.052210 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.052103 2582 scope.go:117] "RemoveContainer" containerID="f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92" Jul 06 21:17:54.053837 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.053813 2582 generic.go:356] "Generic (PLEG): container finished" podID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerID="717ac4962813bed09ad7cc07cbf5aa3eaf704a6dded45d87e146bbed0d19aa6b" exitCode=0 Jul 06 21:17:54.053940 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.053885 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" Jul 06 21:17:54.053991 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.053882 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" event={"ID":"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75","Type":"ContainerDied","Data":"717ac4962813bed09ad7cc07cbf5aa3eaf704a6dded45d87e146bbed0d19aa6b"} Jul 06 21:17:54.054067 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.054043 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" event={"ID":"a34b63b4-ac86-40cf-9b74-fe0cb52f9b75","Type":"ContainerDied","Data":"b86feacc819b490e78995ce48c09e07eea665a21fae255fc93cf2d666f2dffff"} Jul 06 21:17:54.062146 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.062123 2582 scope.go:117] "RemoveContainer" containerID="6d5d5f67bf49bc09f890f0eb54f872bb988515bfa7534f360cdda53d4b03d85f" Jul 06 21:17:54.069893 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.069876 2582 scope.go:117] "RemoveContainer" containerID="f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92" Jul 06 21:17:54.070163 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:17:54.070138 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92\": container with ID starting with f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92 not found: ID does not exist" containerID="f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92" Jul 06 21:17:54.070255 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.070169 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92"} err="failed to get container status \"f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92\": rpc error: code = NotFound desc = could not find container \"f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92\": container with ID starting with f5fc114c9f37d79d55e534d593ae5a82627d87728f5a4f488acb6b14ff2afc92 not found: ID does not exist" Jul 06 21:17:54.070255 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.070187 2582 scope.go:117] "RemoveContainer" containerID="6d5d5f67bf49bc09f890f0eb54f872bb988515bfa7534f360cdda53d4b03d85f" Jul 06 21:17:54.070449 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:17:54.070430 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5d5f67bf49bc09f890f0eb54f872bb988515bfa7534f360cdda53d4b03d85f\": container with ID starting with 6d5d5f67bf49bc09f890f0eb54f872bb988515bfa7534f360cdda53d4b03d85f not found: ID does not exist" containerID="6d5d5f67bf49bc09f890f0eb54f872bb988515bfa7534f360cdda53d4b03d85f" Jul 06 21:17:54.070491 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.070455 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5d5f67bf49bc09f890f0eb54f872bb988515bfa7534f360cdda53d4b03d85f"} err="failed to get container status \"6d5d5f67bf49bc09f890f0eb54f872bb988515bfa7534f360cdda53d4b03d85f\": rpc error: code = NotFound desc = could not find container \"6d5d5f67bf49bc09f890f0eb54f872bb988515bfa7534f360cdda53d4b03d85f\": container with ID starting with 6d5d5f67bf49bc09f890f0eb54f872bb988515bfa7534f360cdda53d4b03d85f not found: ID does not exist" Jul 06 21:17:54.070491 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.070472 2582 scope.go:117] "RemoveContainer" containerID="7ce5f1b6a2ed571a361227480fe620e83b220d4f49edba45790139491655c16b" Jul 06 21:17:54.078206 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.078185 2582 scope.go:117] "RemoveContainer" containerID="717ac4962813bed09ad7cc07cbf5aa3eaf704a6dded45d87e146bbed0d19aa6b" Jul 06 21:17:54.079753 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.079737 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2"] Jul 06 21:17:54.085554 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.085535 2582 scope.go:117] "RemoveContainer" containerID="7ce5f1b6a2ed571a361227480fe620e83b220d4f49edba45790139491655c16b" Jul 06 21:17:54.085828 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:17:54.085808 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ce5f1b6a2ed571a361227480fe620e83b220d4f49edba45790139491655c16b\": container with ID starting with 7ce5f1b6a2ed571a361227480fe620e83b220d4f49edba45790139491655c16b not found: ID does not exist" containerID="7ce5f1b6a2ed571a361227480fe620e83b220d4f49edba45790139491655c16b" Jul 06 21:17:54.085907 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.085835 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ce5f1b6a2ed571a361227480fe620e83b220d4f49edba45790139491655c16b"} err="failed to get container status \"7ce5f1b6a2ed571a361227480fe620e83b220d4f49edba45790139491655c16b\": rpc error: code = NotFound desc = could not find container \"7ce5f1b6a2ed571a361227480fe620e83b220d4f49edba45790139491655c16b\": container with ID starting with 7ce5f1b6a2ed571a361227480fe620e83b220d4f49edba45790139491655c16b not found: ID does not exist" Jul 06 21:17:54.085907 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.085853 2582 scope.go:117] "RemoveContainer" containerID="717ac4962813bed09ad7cc07cbf5aa3eaf704a6dded45d87e146bbed0d19aa6b" Jul 06 21:17:54.086131 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:17:54.086107 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717ac4962813bed09ad7cc07cbf5aa3eaf704a6dded45d87e146bbed0d19aa6b\": container with ID starting with 717ac4962813bed09ad7cc07cbf5aa3eaf704a6dded45d87e146bbed0d19aa6b not found: ID does not exist" containerID="717ac4962813bed09ad7cc07cbf5aa3eaf704a6dded45d87e146bbed0d19aa6b" Jul 06 21:17:54.086182 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.086142 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717ac4962813bed09ad7cc07cbf5aa3eaf704a6dded45d87e146bbed0d19aa6b"} err="failed to get container status \"717ac4962813bed09ad7cc07cbf5aa3eaf704a6dded45d87e146bbed0d19aa6b\": rpc error: code = NotFound desc = could not find container \"717ac4962813bed09ad7cc07cbf5aa3eaf704a6dded45d87e146bbed0d19aa6b\": container with ID starting with 717ac4962813bed09ad7cc07cbf5aa3eaf704a6dded45d87e146bbed0d19aa6b not found: ID does not exist" Jul 06 21:17:54.086835 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.086815 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d7086-predictor-84cb8df759-4cmm2"] Jul 06 21:17:54.100209 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.100188 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn"] Jul 06 21:17:54.102891 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.102870 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn"] Jul 06 21:17:54.576511 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.576471 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": context deadline exceeded" Jul 06 21:17:54.580813 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:54.580791 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d7086-predictor-6848586898-5cvmn" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: i/o timeout" Jul 06 21:17:55.537490 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:55.537455 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" path="/var/lib/kubelet/pods/a34b63b4-ac86-40cf-9b74-fe0cb52f9b75/volumes" Jul 06 21:17:55.539976 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:17:55.537905 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" path="/var/lib/kubelet/pods/ed5be343-7cc7-4eca-8436-1bf8c5dda709/volumes" Jul 06 21:20:33.559250 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:20:33.559175 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 21:20:33.567077 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:20:33.567056 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 21:25:14.268552 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:14.268473 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2"] Jul 06 21:25:14.271091 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:14.268737 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kserve-container" containerID="cri-o://6cab28881f46706b2d426e10cb3bf54e194342c59fb4ac003837bd3378563fe3" gracePeriod=30 Jul 06 21:25:14.271091 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:14.268744 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kube-rbac-proxy" containerID="cri-o://adf695c314b21cf79d92bf2692f5fabbd23d63accf5363f8c6e6700c90b9f3f3" gracePeriod=30 Jul 06 21:25:14.326002 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:14.325968 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf"] Jul 06 21:25:14.326222 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:14.326198 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kserve-container" containerID="cri-o://3694047099c57075f5ad64f9ebd32d71a11e8e87d5b662fb84187f4955bbd19b" gracePeriod=30 Jul 06 21:25:14.326286 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:14.326243 2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kube-rbac-proxy" containerID="cri-o://f68f54e1aac9820e6c2f52243f9edba448bfb59ac11ba191aa05b441c35088be" gracePeriod=30 Jul 06 21:25:14.979307 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:14.979276 2582 generic.go:356] "Generic (PLEG): container finished" podID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerID="f68f54e1aac9820e6c2f52243f9edba448bfb59ac11ba191aa05b441c35088be" exitCode=2 Jul 06 21:25:14.979511 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:14.979346 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" event={"ID":"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958","Type":"ContainerDied","Data":"f68f54e1aac9820e6c2f52243f9edba448bfb59ac11ba191aa05b441c35088be"} Jul 06 21:25:14.980961 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:14.980941 2582 generic.go:356] "Generic (PLEG): container finished" podID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerID="adf695c314b21cf79d92bf2692f5fabbd23d63accf5363f8c6e6700c90b9f3f3" exitCode=2 Jul 06 21:25:14.981055 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:14.981007 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" event={"ID":"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e","Type":"ContainerDied","Data":"adf695c314b21cf79d92bf2692f5fabbd23d63accf5363f8c6e6700c90b9f3f3"} Jul 06 21:25:17.314816 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.314790 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:25:17.396800 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.396707 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-kube-rbac-proxy-sar-config\") pod \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\" (UID: \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\") " Jul 06 21:25:17.396922 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.396837 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5bm8\" (UniqueName: \"kubernetes.io/projected/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-kube-api-access-k5bm8\") pod \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\" (UID: \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\") " Jul 06 21:25:17.396922 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.396874 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-proxy-tls\") pod \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\" (UID: \"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e\") " Jul 06 21:25:17.397077 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.397056 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" (UID: "65e3ebfc-8b2e-4020-b04c-6f294c74ac4e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:25:17.398746 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.398716 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-kube-api-access-k5bm8" (OuterVolumeSpecName: "kube-api-access-k5bm8") pod "65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" (UID: "65e3ebfc-8b2e-4020-b04c-6f294c74ac4e"). InnerVolumeSpecName "kube-api-access-k5bm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:25:17.398880 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.398745 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" (UID: "65e3ebfc-8b2e-4020-b04c-6f294c74ac4e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:25:17.497861 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.497829 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5bm8\" (UniqueName: \"kubernetes.io/projected/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-kube-api-access-k5bm8\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:25:17.497861 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.497855 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:25:17.497861 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.497868 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:25:17.963742 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.963718 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:25:17.996038 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.996008 2582 generic.go:356] "Generic (PLEG): container finished" podID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerID="3694047099c57075f5ad64f9ebd32d71a11e8e87d5b662fb84187f4955bbd19b" exitCode=0 Jul 06 21:25:17.996194 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.996057 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" Jul 06 21:25:17.996194 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.996075 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" event={"ID":"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958","Type":"ContainerDied","Data":"3694047099c57075f5ad64f9ebd32d71a11e8e87d5b662fb84187f4955bbd19b"} Jul 06 21:25:17.996194 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.996150 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf" event={"ID":"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958","Type":"ContainerDied","Data":"e767aa69e64d357b847285b3365d78b6932037e215b10ee949d254d1cf5fbd17"} Jul 06 21:25:17.996321 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.996198 2582 scope.go:117] "RemoveContainer" containerID="f68f54e1aac9820e6c2f52243f9edba448bfb59ac11ba191aa05b441c35088be" Jul 06 21:25:17.997794 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.997771 2582 generic.go:356] "Generic (PLEG): container finished" podID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerID="6cab28881f46706b2d426e10cb3bf54e194342c59fb4ac003837bd3378563fe3" exitCode=0 Jul 06 21:25:17.997909 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.997804 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" event={"ID":"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e","Type":"ContainerDied","Data":"6cab28881f46706b2d426e10cb3bf54e194342c59fb4ac003837bd3378563fe3"} Jul 06 21:25:17.997909 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.997821 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" Jul 06 21:25:17.997909 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:17.997865 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2" event={"ID":"65e3ebfc-8b2e-4020-b04c-6f294c74ac4e","Type":"ContainerDied","Data":"1f5366190fe17fadfab59fa0c09b5848987bf0d46c5555e06348c3549b53eb2d"} Jul 06 21:25:18.004753 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.004735 2582 scope.go:117] "RemoveContainer" containerID="3694047099c57075f5ad64f9ebd32d71a11e8e87d5b662fb84187f4955bbd19b" Jul 06 21:25:18.012431 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.012416 2582 scope.go:117] "RemoveContainer" containerID="f68f54e1aac9820e6c2f52243f9edba448bfb59ac11ba191aa05b441c35088be" Jul 06 21:25:18.012693 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:25:18.012677 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68f54e1aac9820e6c2f52243f9edba448bfb59ac11ba191aa05b441c35088be\": container with ID starting with f68f54e1aac9820e6c2f52243f9edba448bfb59ac11ba191aa05b441c35088be not found: ID does not exist" containerID="f68f54e1aac9820e6c2f52243f9edba448bfb59ac11ba191aa05b441c35088be" Jul 06 21:25:18.012788 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.012704 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68f54e1aac9820e6c2f52243f9edba448bfb59ac11ba191aa05b441c35088be"} err="failed to get container status \"f68f54e1aac9820e6c2f52243f9edba448bfb59ac11ba191aa05b441c35088be\": rpc error: code = NotFound desc = could not find container \"f68f54e1aac9820e6c2f52243f9edba448bfb59ac11ba191aa05b441c35088be\": container with ID starting with f68f54e1aac9820e6c2f52243f9edba448bfb59ac11ba191aa05b441c35088be not found: ID does not exist" Jul 06 21:25:18.012788 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.012725 2582 scope.go:117] "RemoveContainer" containerID="3694047099c57075f5ad64f9ebd32d71a11e8e87d5b662fb84187f4955bbd19b" Jul 06 21:25:18.013097 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:25:18.013077 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3694047099c57075f5ad64f9ebd32d71a11e8e87d5b662fb84187f4955bbd19b\": container with ID starting with 3694047099c57075f5ad64f9ebd32d71a11e8e87d5b662fb84187f4955bbd19b not found: ID does not exist" containerID="3694047099c57075f5ad64f9ebd32d71a11e8e87d5b662fb84187f4955bbd19b" Jul 06 21:25:18.013151 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.013113 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3694047099c57075f5ad64f9ebd32d71a11e8e87d5b662fb84187f4955bbd19b"} err="failed to get container status \"3694047099c57075f5ad64f9ebd32d71a11e8e87d5b662fb84187f4955bbd19b\": rpc error: code = NotFound desc = could not find container \"3694047099c57075f5ad64f9ebd32d71a11e8e87d5b662fb84187f4955bbd19b\": container with ID starting with 3694047099c57075f5ad64f9ebd32d71a11e8e87d5b662fb84187f4955bbd19b not found: ID does not exist" Jul 06 21:25:18.013151 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.013134 2582 scope.go:117] "RemoveContainer" containerID="adf695c314b21cf79d92bf2692f5fabbd23d63accf5363f8c6e6700c90b9f3f3" Jul 06 21:25:18.019123 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.019105 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2"] Jul 06 21:25:18.020356 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.020345 2582 scope.go:117] "RemoveContainer" containerID="6cab28881f46706b2d426e10cb3bf54e194342c59fb4ac003837bd3378563fe3" Jul 06 21:25:18.024861 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.024841 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32e32-predictor-5fb6b9c6c8-2hsv2"] Jul 06 21:25:18.027525 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.027510 2582 scope.go:117] "RemoveContainer" containerID="adf695c314b21cf79d92bf2692f5fabbd23d63accf5363f8c6e6700c90b9f3f3" Jul 06 21:25:18.027804 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:25:18.027748 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf695c314b21cf79d92bf2692f5fabbd23d63accf5363f8c6e6700c90b9f3f3\": container with ID starting with adf695c314b21cf79d92bf2692f5fabbd23d63accf5363f8c6e6700c90b9f3f3 not found: ID does not exist" containerID="adf695c314b21cf79d92bf2692f5fabbd23d63accf5363f8c6e6700c90b9f3f3" Jul 06 21:25:18.027892 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.027815 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf695c314b21cf79d92bf2692f5fabbd23d63accf5363f8c6e6700c90b9f3f3"} err="failed to get container status \"adf695c314b21cf79d92bf2692f5fabbd23d63accf5363f8c6e6700c90b9f3f3\": rpc error: code = NotFound desc = could not find container \"adf695c314b21cf79d92bf2692f5fabbd23d63accf5363f8c6e6700c90b9f3f3\": container with ID starting with adf695c314b21cf79d92bf2692f5fabbd23d63accf5363f8c6e6700c90b9f3f3 not found: ID does not exist" Jul 06 21:25:18.027892 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.027836 2582 scope.go:117] "RemoveContainer" containerID="6cab28881f46706b2d426e10cb3bf54e194342c59fb4ac003837bd3378563fe3" Jul 06 21:25:18.028088 ip-10-0-142-142 kubenswrapper[2582]: E0706 21:25:18.028070 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cab28881f46706b2d426e10cb3bf54e194342c59fb4ac003837bd3378563fe3\": container with ID starting with 6cab28881f46706b2d426e10cb3bf54e194342c59fb4ac003837bd3378563fe3 not found: ID does not exist" containerID="6cab28881f46706b2d426e10cb3bf54e194342c59fb4ac003837bd3378563fe3" Jul 06 21:25:18.028122 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.028095 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cab28881f46706b2d426e10cb3bf54e194342c59fb4ac003837bd3378563fe3"} err="failed to get container status \"6cab28881f46706b2d426e10cb3bf54e194342c59fb4ac003837bd3378563fe3\": rpc error: code = NotFound desc = could not find container \"6cab28881f46706b2d426e10cb3bf54e194342c59fb4ac003837bd3378563fe3\": container with ID starting with 6cab28881f46706b2d426e10cb3bf54e194342c59fb4ac003837bd3378563fe3 not found: ID does not exist" Jul 06 21:25:18.103654 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.103626 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-proxy-tls\") pod \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\" (UID: \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\") " Jul 06 21:25:18.103810 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.103688 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x6kr\" (UniqueName: \"kubernetes.io/projected/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-kube-api-access-9x6kr\") pod \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\" (UID: \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\") " Jul 06 21:25:18.103810 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.103737 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-kube-rbac-proxy-sar-config\") pod \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\" (UID: \"29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958\") " Jul 06 21:25:18.104161 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.104131 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" (UID: "29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:25:18.105739 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.105716 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" (UID: "29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:25:18.105836 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.105784 2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-kube-api-access-9x6kr" (OuterVolumeSpecName: "kube-api-access-9x6kr") pod "29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" (UID: "29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958"). InnerVolumeSpecName "kube-api-access-9x6kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:25:18.204303 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.204254 2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:25:18.204303 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.204296 2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-proxy-tls\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:25:18.204303 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.204306 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x6kr\" (UniqueName: \"kubernetes.io/projected/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958-kube-api-access-9x6kr\") on node \"ip-10-0-142-142.ec2.internal\" DevicePath \"\"" Jul 06 21:25:18.323569 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.323538 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf"] Jul 06 21:25:18.325505 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:18.325485 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-32e32-predictor-98848c587-cfphf"] Jul 06 21:25:19.543225 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:19.543192 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" path="/var/lib/kubelet/pods/29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958/volumes" Jul 06 21:25:19.543604 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:19.543590 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" path="/var/lib/kubelet/pods/65e3ebfc-8b2e-4020-b04c-6f294c74ac4e/volumes" Jul 06 21:25:33.583274 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:33.583250 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 21:25:33.591656 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:33.591637 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 21:25:47.329832 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.329794 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8gcgd/must-gather-wlbjq"] Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330188 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kserve-container" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330203 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kserve-container" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330211 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kserve-container" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330216 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kserve-container" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330225 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kserve-container" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330231 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kserve-container" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330238 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kube-rbac-proxy" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330246 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kube-rbac-proxy" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330258 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kube-rbac-proxy" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330264 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kube-rbac-proxy" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330277 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kube-rbac-proxy" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330283 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kube-rbac-proxy" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330290 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kube-rbac-proxy" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330295 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kube-rbac-proxy" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330300 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kserve-container" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330305 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kserve-container" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330310 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kube-rbac-proxy" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330314 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kube-rbac-proxy" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330322 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kserve-container" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330327 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kserve-container" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330333 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kserve-container" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330337 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kserve-container" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330343 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kube-rbac-proxy" Jul 06 21:25:47.330370 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330348 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kube-rbac-proxy" Jul 06 21:25:47.331154 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330401 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kserve-container" Jul 06 21:25:47.331154 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330415 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kserve-container" Jul 06 21:25:47.331154 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330423 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34b63b4-ac86-40cf-9b74-fe0cb52f9b75" containerName="kube-rbac-proxy" Jul 06 21:25:47.331154 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330434 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kserve-container" Jul 06 21:25:47.331154 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330443 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kserve-container" Jul 06 21:25:47.331154 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330451 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed5be343-7cc7-4eca-8436-1bf8c5dda709" containerName="kube-rbac-proxy" Jul 06 21:25:47.331154 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330458 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae36521-e6ea-446b-892c-51cc9ec16d81" containerName="kube-rbac-proxy" Jul 06 21:25:47.331154 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330467 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b44663-269b-4ead-bbd4-09f1909c4e9b" containerName="kube-rbac-proxy" Jul 06 21:25:47.331154 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330476 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kube-rbac-proxy" Jul 06 21:25:47.331154 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330482 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kube-rbac-proxy" Jul 06 21:25:47.331154 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330487 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="29bd7197-a4d2-4c4b-9cbc-83f4a2f2d958" containerName="kserve-container" Jul 06 21:25:47.331154 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.330492 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="65e3ebfc-8b2e-4020-b04c-6f294c74ac4e" containerName="kserve-container" Jul 06 21:25:47.333635 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.333615 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8gcgd/must-gather-wlbjq" Jul 06 21:25:47.336528 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.336508 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8gcgd\"/\"openshift-service-ca.crt\"" Jul 06 21:25:47.336660 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.336555 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8gcgd\"/\"kube-root-ca.crt\"" Jul 06 21:25:47.336660 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.336648 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8gcgd\"/\"default-dockercfg-vxddg\"" Jul 06 21:25:47.344943 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.344922 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8gcgd/must-gather-wlbjq"] Jul 06 21:25:47.434296 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.434257 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95509861-b118-489a-ae1d-9c683f2b409f-must-gather-output\") pod \"must-gather-wlbjq\" (UID: \"95509861-b118-489a-ae1d-9c683f2b409f\") " pod="openshift-must-gather-8gcgd/must-gather-wlbjq" Jul 06 21:25:47.434296 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.434301 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s27cb\" (UniqueName: \"kubernetes.io/projected/95509861-b118-489a-ae1d-9c683f2b409f-kube-api-access-s27cb\") pod \"must-gather-wlbjq\" (UID: \"95509861-b118-489a-ae1d-9c683f2b409f\") " pod="openshift-must-gather-8gcgd/must-gather-wlbjq" Jul 06 21:25:47.534709 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.534672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s27cb\" (UniqueName: \"kubernetes.io/projected/95509861-b118-489a-ae1d-9c683f2b409f-kube-api-access-s27cb\") pod \"must-gather-wlbjq\" (UID: \"95509861-b118-489a-ae1d-9c683f2b409f\") " pod="openshift-must-gather-8gcgd/must-gather-wlbjq" Jul 06 21:25:47.534925 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.534743 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95509861-b118-489a-ae1d-9c683f2b409f-must-gather-output\") pod \"must-gather-wlbjq\" (UID: \"95509861-b118-489a-ae1d-9c683f2b409f\") " pod="openshift-must-gather-8gcgd/must-gather-wlbjq" Jul 06 21:25:47.535046 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.535029 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95509861-b118-489a-ae1d-9c683f2b409f-must-gather-output\") pod \"must-gather-wlbjq\" (UID: \"95509861-b118-489a-ae1d-9c683f2b409f\") " pod="openshift-must-gather-8gcgd/must-gather-wlbjq" Jul 06 21:25:47.544430 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.544408 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s27cb\" (UniqueName: \"kubernetes.io/projected/95509861-b118-489a-ae1d-9c683f2b409f-kube-api-access-s27cb\") pod \"must-gather-wlbjq\" (UID: \"95509861-b118-489a-ae1d-9c683f2b409f\") " pod="openshift-must-gather-8gcgd/must-gather-wlbjq" Jul 06 21:25:47.649709 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.649678 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8gcgd/must-gather-wlbjq" Jul 06 21:25:47.774718 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.774691 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8gcgd/must-gather-wlbjq"] Jul 06 21:25:47.776944 ip-10-0-142-142 kubenswrapper[2582]: W0706 21:25:47.776897 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95509861_b118_489a_ae1d_9c683f2b409f.slice/crio-bf0394a7fe85ad25c7a074e2ddf8aa03d49a0908998c992de72aff4029dc43ae WatchSource:0}: Error finding container bf0394a7fe85ad25c7a074e2ddf8aa03d49a0908998c992de72aff4029dc43ae: Status 404 returned error can't find the container with id bf0394a7fe85ad25c7a074e2ddf8aa03d49a0908998c992de72aff4029dc43ae Jul 06 21:25:47.778817 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:47.778802 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 21:25:48.123292 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:48.123258 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8gcgd/must-gather-wlbjq" event={"ID":"95509861-b118-489a-ae1d-9c683f2b409f","Type":"ContainerStarted","Data":"bf0394a7fe85ad25c7a074e2ddf8aa03d49a0908998c992de72aff4029dc43ae"} Jul 06 21:25:49.132509 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:49.132463 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8gcgd/must-gather-wlbjq" event={"ID":"95509861-b118-489a-ae1d-9c683f2b409f","Type":"ContainerStarted","Data":"7012502cf3cf2901aeb283272eaae9b1c80bd2fd3edd80f68f9dd39e89f37371"} Jul 06 21:25:49.133048 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:49.132514 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8gcgd/must-gather-wlbjq" event={"ID":"95509861-b118-489a-ae1d-9c683f2b409f","Type":"ContainerStarted","Data":"f55cf9c1c2a5a41345603094bf0d92883442ba25522621db0b1606d696eb8fb3"} Jul 06 21:25:49.152808 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:49.152714 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8gcgd/must-gather-wlbjq" podStartSLOduration=1.107966733 podStartE2EDuration="2.152698937s" totalImagesPullingTime="1.044732204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:25:49.151573528 +0000 UTC m=+3016.237128722" watchObservedRunningTime="2026-07-06 21:25:49.152698937 +0000 UTC m=+3016.238254129" Jul 06 21:25:50.290249 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:50.290223 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5zpwl_63a48280-235d-4f85-a741-ff9d9c3aaa3b/global-pull-secret-syncer/0.log" Jul 06 21:25:50.404490 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:50.404459 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-crf7d_d909edb1-81aa-4c46-8b90-18691df9680c/konnectivity-agent/0.log" Jul 06 21:25:50.523032 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:50.523003 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-142.ec2.internal_2150a52d3ba57389650b4becb0fa9935/haproxy/0.log" Jul 06 21:25:54.426912 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:54.426796 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-7fkc2_54b52e62-b5cd-4d43-adcf-32649b70e2d6/cluster-monitoring-operator/0.log" Jul 06 21:25:54.453378 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:54.453328 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-rxxbj_c366d3eb-8e1c-4934-9039-50bc149983eb/kube-state-metrics/0.log" Jul 06 21:25:54.476393 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:54.476361 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-rxxbj_c366d3eb-8e1c-4934-9039-50bc149983eb/kube-rbac-proxy-main/0.log" Jul 06 21:25:54.506041 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:54.506013 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-rxxbj_c366d3eb-8e1c-4934-9039-50bc149983eb/kube-rbac-proxy-self/0.log" Jul 06 21:25:54.542962 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:54.542935 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f6567c9b6-d6gt6_194f7868-6ebd-4220-a09d-d21f124c5c2d/metrics-server/0.log" Jul 06 21:25:54.580481 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:54.580455 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-ff52b_83cba701-7e6b-4e88-a4fb-2d984b3fa9b4/monitoring-plugin/0.log" Jul 06 21:25:54.611351 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:54.611325 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7bv7x_91ad3522-11b2-4d4d-b6a5-d913f4b24933/node-exporter/0.log" Jul 06 21:25:54.632923 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:54.632887 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7bv7x_91ad3522-11b2-4d4d-b6a5-d913f4b24933/kube-rbac-proxy/0.log" Jul 06 21:25:54.654927 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:54.654894 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7bv7x_91ad3522-11b2-4d4d-b6a5-d913f4b24933/init-textfile/0.log" Jul 06 21:25:55.134559 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:55.134525 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-zc2gl_a8f6bf9b-d82a-4800-9024-9ba87b46f2fb/prometheus-operator/0.log" Jul 06 21:25:55.154789 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:55.154722 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-zc2gl_a8f6bf9b-d82a-4800-9024-9ba87b46f2fb/kube-rbac-proxy/0.log" Jul 06 21:25:55.182784 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:55.182729 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-qtgmv_a81bf929-9dbc-4714-8ace-3c27e9c96b30/prometheus-operator-admission-webhook/0.log" Jul 06 21:25:56.721110 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:56.721072 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-p4fbr_a472ef31-0009-4bc8-97f2-72ce4847739d/networking-console-plugin/0.log" Jul 06 21:25:57.706664 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:57.706638 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bbf69ddd9-nmzsq_758e406a-bd93-4cc2-a19e-3b50000c7889/console/0.log" Jul 06 21:25:57.744152 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:57.744121 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-gvh6p_9f0b356c-3738-4e8e-a2f4-8b159924540a/download-server/0.log" Jul 06 21:25:58.279085 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.279055 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-mzdjm_03395407-4e74-4b69-a2df-7f86f312db21/volume-data-source-validator/0.log" Jul 06 21:25:58.325208 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.325172 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh"] Jul 06 21:25:58.330850 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.330823 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.342449 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.342418 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh"] Jul 06 21:25:58.437450 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.437419 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v922t\" (UniqueName: \"kubernetes.io/projected/005bc23d-22d5-4b6e-8d7e-7b24703777f7-kube-api-access-v922t\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.437641 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.437477 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/005bc23d-22d5-4b6e-8d7e-7b24703777f7-podres\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.437641 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.437502 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/005bc23d-22d5-4b6e-8d7e-7b24703777f7-sys\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.437641 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.437543 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/005bc23d-22d5-4b6e-8d7e-7b24703777f7-lib-modules\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.437641 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.437592 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/005bc23d-22d5-4b6e-8d7e-7b24703777f7-proc\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.538938 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.538832 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v922t\" (UniqueName: \"kubernetes.io/projected/005bc23d-22d5-4b6e-8d7e-7b24703777f7-kube-api-access-v922t\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.538938 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.538900 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/005bc23d-22d5-4b6e-8d7e-7b24703777f7-podres\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.538938 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.538926 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/005bc23d-22d5-4b6e-8d7e-7b24703777f7-sys\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.539422 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.538961 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/005bc23d-22d5-4b6e-8d7e-7b24703777f7-lib-modules\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.539422 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.538999 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/005bc23d-22d5-4b6e-8d7e-7b24703777f7-proc\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.539422 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.539120 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/005bc23d-22d5-4b6e-8d7e-7b24703777f7-proc\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.539422 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.539238 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/005bc23d-22d5-4b6e-8d7e-7b24703777f7-podres\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.539422 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.539284 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/005bc23d-22d5-4b6e-8d7e-7b24703777f7-sys\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.539617 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.539425 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/005bc23d-22d5-4b6e-8d7e-7b24703777f7-lib-modules\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.550501 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.550470 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v922t\" (UniqueName: \"kubernetes.io/projected/005bc23d-22d5-4b6e-8d7e-7b24703777f7-kube-api-access-v922t\") pod \"perf-node-gather-daemonset-ftbfh\" (UID: \"005bc23d-22d5-4b6e-8d7e-7b24703777f7\") " pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.644056 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.644021 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:58.815920 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:58.815784 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh"] Jul 06 21:25:59.195855 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:59.195817 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" event={"ID":"005bc23d-22d5-4b6e-8d7e-7b24703777f7","Type":"ContainerStarted","Data":"f226a7d9e532996356b1bb41c5887196014bf6cd3fabe482ab8c68689c60b7d8"} Jul 06 21:25:59.196074 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:59.195881 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" event={"ID":"005bc23d-22d5-4b6e-8d7e-7b24703777f7","Type":"ContainerStarted","Data":"6f9c1b8d188465cc13608fc7366201537c3e63543301a495327d1cdf6fbec939"} Jul 06 21:25:59.196424 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:59.196222 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:25:59.214653 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:59.214609 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" podStartSLOduration=1.214594503 podStartE2EDuration="1.214594503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:25:59.213421232 +0000 UTC m=+3026.298976426" watchObservedRunningTime="2026-07-06 21:25:59.214594503 +0000 UTC m=+3026.300149694" Jul 06 21:25:59.381362 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:59.381333 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t8qhq_5c07f3a3-0155-4529-a90d-d5dbdacf7b15/dns/0.log" Jul 06 21:25:59.405124 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:59.405100 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t8qhq_5c07f3a3-0155-4529-a90d-d5dbdacf7b15/kube-rbac-proxy/0.log" Jul 06 21:25:59.431032 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:25:59.431004 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nsbpk_e609c626-c0c6-4b01-9640-e63a163c6c96/dns-node-resolver/0.log" Jul 06 21:26:00.017863 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:00.017819 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fzclm_a203ceeb-e878-49c1-b1e1-ff93471aa9fb/node-ca/0.log" Jul 06 21:26:01.362720 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:01.362663 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b7qtk_796f6480-1067-43ca-b1f9-eff19e1a8720/serve-healthcheck-canary/0.log" Jul 06 21:26:01.920968 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:01.920932 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lcm6v_c111398d-6c9f-4d44-9b46-95fe1fdf4248/kube-rbac-proxy/0.log" Jul 06 21:26:01.942037 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:01.942014 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lcm6v_c111398d-6c9f-4d44-9b46-95fe1fdf4248/exporter/0.log" Jul 06 21:26:01.963305 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:01.963280 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lcm6v_c111398d-6c9f-4d44-9b46-95fe1fdf4248/extractor/0.log" Jul 06 21:26:04.066978 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:04.066955 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-zcnd4_8dbea9d6-ed55-45cd-93bd-e5541f8374b5/server/0.log" Jul 06 21:26:04.292378 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:04.292352 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-9ctjx_3ad1a5b4-dbce-4b1f-b2cb-e40c48bbd61f/seaweedfs/0.log" Jul 06 21:26:05.213193 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:05.213168 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8gcgd/perf-node-gather-daemonset-ftbfh" Jul 06 21:26:08.547694 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:08.547663 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-nq9k6_22500c6c-3d10-49f4-be87-5e9efea17085/migrator/0.log" Jul 06 21:26:08.571199 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:08.571163 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-nq9k6_22500c6c-3d10-49f4-be87-5e9efea17085/graceful-termination/0.log" Jul 06 21:26:09.867881 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:09.867852 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6jqmg_2a6c9a93-1c46-445b-a246-7251d7f007c9/kube-multus/0.log" Jul 06 21:26:10.238139 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:10.238105 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqtv5_9d18cb24-9db4-42fe-93b8-a47b2f26a78f/kube-multus-additional-cni-plugins/0.log" Jul 06 21:26:10.261647 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:10.261617 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqtv5_9d18cb24-9db4-42fe-93b8-a47b2f26a78f/egress-router-binary-copy/0.log" Jul 06 21:26:10.283373 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:10.283349 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqtv5_9d18cb24-9db4-42fe-93b8-a47b2f26a78f/cni-plugins/0.log" Jul 06 21:26:10.304267 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:10.304244 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqtv5_9d18cb24-9db4-42fe-93b8-a47b2f26a78f/bond-cni-plugin/0.log" Jul 06 21:26:10.325466 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:10.325443 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqtv5_9d18cb24-9db4-42fe-93b8-a47b2f26a78f/routeoverride-cni/0.log" Jul 06 21:26:10.349561 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:10.349534 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqtv5_9d18cb24-9db4-42fe-93b8-a47b2f26a78f/whereabouts-cni-bincopy/0.log" Jul 06 21:26:10.370839 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:10.370799 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqtv5_9d18cb24-9db4-42fe-93b8-a47b2f26a78f/whereabouts-cni/0.log" Jul 06 21:26:10.513216 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:10.513138 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d26rt_4d976ba8-6cb4-4882-9f06-e9c1d62cc475/network-metrics-daemon/0.log" Jul 06 21:26:10.533901 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:10.533877 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d26rt_4d976ba8-6cb4-4882-9f06-e9c1d62cc475/kube-rbac-proxy/0.log" Jul 06 21:26:11.563894 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:11.563862 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-controller/0.log" Jul 06 21:26:11.581156 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:11.581131 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/1.log" Jul 06 21:26:11.581315 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:11.581131 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovn-acl-logging/0.log" Jul 06 21:26:11.601663 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:11.601639 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/kube-rbac-proxy-node/0.log" Jul 06 21:26:11.622926 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:11.622903 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/kube-rbac-proxy-ovn-metrics/0.log" Jul 06 21:26:11.643486 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:11.643443 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/northd/0.log" Jul 06 21:26:11.663868 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:11.663845 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/nbdb/0.log" Jul 06 21:26:11.687576 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:11.687556 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/sbdb/0.log" Jul 06 21:26:11.733356 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:11.733310 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crszk_82ffacd0-c326-48d3-8e22-be96a8e06e17/ovnkube-controller/0.log" Jul 06 21:26:13.000376 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:13.000337 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-sxdx7_2fb01258-3ad2-4383-b313-3af4d4a40c54/network-check-target-container/0.log" Jul 06 21:26:13.946202 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:13.946176 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5jb4d_3268a9a6-5687-440f-91a0-5916a32aa1a5/iptables-alerter/0.log" Jul 06 21:26:14.658531 ip-10-0-142-142 kubenswrapper[2582]: I0706 21:26:14.658446 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-jwpp2_d8a0e3f3-d1d1-4732-81a9-035c2babecb9/tuned/0.log"