Sep 15 16:26:48.860027 ip-10-0-134-77 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 16:26:49.264680 ip-10-0-134-77 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 16:26:49.264680 ip-10-0-134-77 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 16:26:49.264680 ip-10-0-134-77 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 16:26:49.264680 ip-10-0-134-77 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 16:26:49.264680 ip-10-0-134-77 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 16:26:49.266631 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.266545    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 16:26:49.273454 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273432    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:49.273454 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273449    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:49.273454 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273453    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:49.273454 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273457    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:49.273454 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273460    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273464    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273469    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273474    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273477    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273481    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273484    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273487    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273490    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273492    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273495    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273498    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273501    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273503    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273506    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273509    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273512    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273514    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273517    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273520    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:49.273642 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273522    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273525    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273528    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273530    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273534    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273536    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273539    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273549    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273552    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273555    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273558    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273560    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273563    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273565    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273568    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273571    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273573    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273576    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273578    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273581    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:49.274195 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273584    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273587    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273589    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273592    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273595    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273597    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273599    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273602    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273605    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273607    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273610    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273613    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273615    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273618    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273622    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273626    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273629    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273631    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273634    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273636    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:49.274685 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273639    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273641    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273644    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273647    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273649    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273652    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273656    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273659    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273661    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273664    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273667    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273672    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273676    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273679    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273682    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273691    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273694    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273697    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273700    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:49.275180 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273703    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273721    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273724    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273727    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273729    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273732    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273734    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273737    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273739    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273743    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273746    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273748    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273751    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273753    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273756    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.273758    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274492    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274499    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274502    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:49.275673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274506    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274509    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274512    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274514    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274517    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274520    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274523    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274526    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274528    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274531    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274534    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274537    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274540    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274542    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274545    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274548    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274551    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274553    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274556    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274558    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:49.276163 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274562    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274565    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274568    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274571    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274573    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274576    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274578    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274581    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274583    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274586    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274591    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274595    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274598    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274601    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274605    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274608    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274611    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274614    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274617    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:49.276698 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274620    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274623    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274626    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274629    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274631    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274634    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274636    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274639    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274642    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274644    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274647    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274650    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274652    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274656    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274659    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274661    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274664    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274667    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274670    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:49.277198 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274672    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274675    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274677    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274680    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274683    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274685    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274688    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274691    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274693    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274696    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274699    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274701    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274722    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274725    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274728    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274731    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274733    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274736    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274738    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274741    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274744    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:49.277660 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274748    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274751    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274754    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274757    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274760    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274762    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274765    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274768    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274771    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274773    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274776    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274779    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274783    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274786    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274789    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274792    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.274795    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275170    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275175    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:49.278186 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275178    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275181    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275184    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275187    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275189    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275192    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275195    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275197    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275200    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275203    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275206    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275208    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275211    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275214    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275217    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275219    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275222    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275225    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275228    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275230    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:49.278656 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275233    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275235    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275238    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275240    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275243    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275245    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275248    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275250    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275252    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275255    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275258    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275260    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275263    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275267    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275270    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275272    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275275    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275277    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275280    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:49.279151 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275283    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275287    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275290    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275293    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275296    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275299    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275302    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275304    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275308    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275312    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275315    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275317    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275320    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275323    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275326    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275328    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275331    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275334    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275336    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:49.279610 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275339    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275342    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275345    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275348    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275351    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275354    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275357    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275360    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275362    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275365    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275368    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275370    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275372    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275375    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275377    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275380    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275385    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275388    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275391    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275393    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:49.280090 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275396    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275398    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275403    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275406    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275409    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275411    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275414    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275416    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275419    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275421    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275424    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275427    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275429    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275432    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275435    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275437    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275440    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275443    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.275445    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275523    2565 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275559    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 16:26:49.280576 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275565    2565 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275570    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275574    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275578    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275582    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275586    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275590    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275594    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275598    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275601    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275604    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275607    2565 flags.go:64] FLAG: --cgroup-root=""
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275610    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275614    2565 flags.go:64] FLAG: --client-ca-file=""
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275618    2565 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275620    2565 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275626    2565 flags.go:64] FLAG: --cluster-domain=""
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275629    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275632    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275635    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275638    2565 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275642    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275645    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 16:26:49.281121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275648    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275652    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275654    2565 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275657    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275660    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275664    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275666    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275670    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275673    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275676    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275679    2565 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275682    2565 flags.go:64] FLAG: --enable-server="true"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275685    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275690    2565 flags.go:64] FLAG: --event-burst="100"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275692    2565 flags.go:64] FLAG: --event-qps="50"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275695    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275700    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275703    2565 flags.go:64] FLAG: --eviction-hard=""
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275720    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275723    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275726    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275729    2565 flags.go:64] FLAG: --eviction-soft=""
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275734    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275737    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275741    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 16:26:49.281728 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275744    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275747    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275749    2565 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275752    2565 flags.go:64] FLAG: --feature-gates=""
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275756    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275759    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275762    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275765    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275768    2565 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275771    2565 flags.go:64] FLAG: --help="false"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275774    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275777    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275780    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275783    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275786    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275790    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275793    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275796    2565 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275798    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275801    2565 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275804    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275807    2565 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275810    2565 flags.go:64] FLAG: --kube-reserved=""
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275814    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 16:26:49.282377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275817    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275820    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275823    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275825    2565 flags.go:64] FLAG: --lock-file=""
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275828    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275832    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275835    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275841    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275844    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275847    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275850    2565 flags.go:64] FLAG: --logging-format="text"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275853    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275856    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275859    2565 flags.go:64] FLAG: --manifest-url=""
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275862    2565 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275866    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275869    2565 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275873    2565 flags.go:64] FLAG: --max-pods="110"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275876    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275879    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275882    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275885    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275888    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275891    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275893    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 16:26:49.283062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275901    2565 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275904    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275907    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275910    2565 flags.go:64] FLAG: --pod-cidr=""
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275913    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275918    2565 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275922    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275925    2565 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275928    2565 flags.go:64] FLAG: --port="10250"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275931    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275934    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0615f6e1fbad1e801"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275937    2565 flags.go:64] FLAG: --qos-reserved=""
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275940    2565 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275944    2565 flags.go:64] FLAG: --register-node="true"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275947    2565 flags.go:64] FLAG: --register-with-taints=""
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275951    2565 flags.go:64] FLAG: --registry-burst="10"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275954    2565 flags.go:64] FLAG: --registry-qps="5"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275957    2565 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275960    2565 flags.go:64] FLAG: --reserved-memory=""
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275964    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275967    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275969    2565 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275972    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275975    2565 flags.go:64] FLAG: --runonce="false"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275978    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 16:26:49.283682 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275981    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275984    2565 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275986    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275989    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275992    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275995    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.275998    2565 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276001    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276004    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276007    2565 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276010    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276013    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276016    2565 flags.go:64] FLAG: --system-cgroups=""
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276020    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276025    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276028    2565 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276030    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276035    2565 flags.go:64] FLAG: --tls-min-version=""
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276038    2565 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276041    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276045    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276048    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276051    2565 flags.go:64] FLAG: --v="2"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276054    2565 flags.go:64] FLAG: --version="false"
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276059    2565 flags.go:64] FLAG: --vmodule=""
Sep 15 16:26:49.284379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276063    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.276066    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276179    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276183    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276187    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276190    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276192    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276195    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276198    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276201    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276204    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276206    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276209    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276211    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276214    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276217    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276219    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276222    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276225    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276227    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:49.285043 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276233    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276237    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276240    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276244    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276248    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276251    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276254    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276258    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276261    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276264    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276267    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276270    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276274    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276277    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276280    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276282    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276285    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276288    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276290    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276292    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:49.285531 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276295    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276298    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276300    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276303    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276306    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276308    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276311    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276314    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276316    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276319    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276321    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276324    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276328    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276330    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276333    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276336    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276339    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276341    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276344    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276348    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:49.286074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276350    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276354    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276358    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276361    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276363    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276366    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276369    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276372    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276375    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276377    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276379    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276382    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276385    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276387    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276390    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276392    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276395    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276397    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276400    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:49.286577 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276403    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276405    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276408    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276410    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276413    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276418    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276420    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276423    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276425    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276428    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276431    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276433    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276437    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276440    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276442    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276445    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276457    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276460    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276462    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276465    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:49.287074 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276468    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.276471    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.277186    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.284810    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.284828    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.284845    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.284851    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284909    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284914    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284917    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284920    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284924    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284927    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284930    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:49.287572 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284933    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284936    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284939    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284941    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284944    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284947    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284949    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284952    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284955    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284957    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284960    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284962    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284965    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284968    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284970    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284973    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284976    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284979    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284982    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284985    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:49.287936 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284988    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284990    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284994    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284996    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.284999    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285002    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285005    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285007    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285010    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285012    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285015    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285019    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285023    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285026    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285029    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285032    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285035    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285037    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285041    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:49.288448 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285043    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285047    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285050    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285053    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285056    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285059    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285061    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285064    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285067    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285069    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285072    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285075    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285078    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285080    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285083    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285087    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285089    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285092    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285095    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285097    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:49.288998 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285100    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285102    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285105    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285108    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285110    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285113    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285115    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285118    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285123    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285127    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285130    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285132    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285135    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285137    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285141    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285144    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285147    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285150    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285153    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:49.289528 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285155    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285159    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285161    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285164    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285167    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285170    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285172    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285175    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285178    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285180    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285183    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285186    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285189    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285191    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.285196    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 16:26:49.290004 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285313    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285318    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285321    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285324    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285327    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285330    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285332    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285335    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285338    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285340    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285343    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285346    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285349    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285351    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285354    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285357    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285359    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285362    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285365    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285367    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:49.290384 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285370    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285372    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285375    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285378    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285381    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285383    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285386    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285389    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285392    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285394    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285397    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285399    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285402    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285404    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285407    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285409    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285412    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285414    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285417    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285419    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:49.290884 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285422    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285424    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285428    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285430    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285433    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285435    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285438    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285440    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285443    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285445    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285448    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285450    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285453    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285455    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285459    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285463    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285466    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285469    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285472    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285475    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:49.291374 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285478    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285480    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285483    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285486    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285488    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285491    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285493    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285496    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285499    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285501    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285503    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285506    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285510    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285513    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285516    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285519    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285522    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285525    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285528    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:49.291866 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285530    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285538    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285541    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285544    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285547    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285549    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285552    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285555    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285558    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285560    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285563    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285566    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285569    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285571    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285574    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285576    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285579    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285582    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285585    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:49.292319 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:49.285588    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:49.292857 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.285593    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 16:26:49.292857 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.285694    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 16:26:49.292857 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.288134    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 16:26:49.292857 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.289256    2565 server.go:1019] "Starting client certificate rotation"
Sep 15 16:26:49.292857 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.289356    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 16:26:49.292857 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.289392    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 16:26:49.292857 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.292847    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 16:26:49.293980 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.293963    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 16:26:49.304397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.304380    2565 log.go:25] "Validated CRI v1 runtime API"
Sep 15 16:26:49.307799 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.307784    2565 log.go:25] "Validated CRI v1 image API"
Sep 15 16:26:49.308637 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.308620    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 16:26:49.311344 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.311317    2565 fs.go:135] Filesystem UUIDs: map[329448a5-1b49-4996-80cc-b772a9134bbc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c30fb543-1f09-4f8b-a5f0-32d582ac864f:/dev/nvme0n1p3]
Sep 15 16:26:49.311403 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.311344    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 16:26:49.316811 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.316673    2565 manager.go:216] Machine: {Timestamp:2026-09-15 16:26:49.315256166 +0000 UTC m=+0.359641889 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3076483 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cac47a23e05e1065481149a747a09 SystemUUID:ec2cac47-a23e-05e1-0654-81149a747a09 BootID:6349f1c6-29c1-4b7b-8cfb-122b6aac0ae8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:03:dd:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:03:dd:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:76:d9:c4:1f:7b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 16:26:49.316811 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.316803    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 16:26:49.316943 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.316909    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 16:26:49.317967 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.317948    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 16:26:49.320360 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.320331    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 16:26:49.320490 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.320363    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-77.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 16:26:49.321582 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.321572    2565 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 16:26:49.321622 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.321586    2565 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 16:26:49.321622 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.321593    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 16:26:49.322539 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.322530    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 16:26:49.323699 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.323689    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 16:26:49.327218 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.327206    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 16:26:49.327339 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.327329    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 16:26:49.327375 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.327359    2565 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 16:26:49.327375 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.327374    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 16:26:49.327435 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.327389    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 16:26:49.327435 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.327397    2565 kubelet.go:398] "Adding apiserver pod source"
Sep 15 16:26:49.327435 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.327406    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 16:26:49.327819 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.327808    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 16:26:49.327879 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.327828    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 16:26:49.331014 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.330993    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 16:26:49.333097 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.333083    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 16:26:49.333167 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.333113    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 16:26:49.335117 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335099    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 16:26:49.335117 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335117    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 16:26:49.335225 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335123    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 16:26:49.335225 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335129    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 16:26:49.335225 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335137    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 16:26:49.335225 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335142    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 16:26:49.335225 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335155    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 16:26:49.335225 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335160    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 16:26:49.335225 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335167    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 16:26:49.335225 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335172    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 16:26:49.335225 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335180    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 16:26:49.335225 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335185    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 16:26:49.335225 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335207    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 16:26:49.335225 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.335212    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 16:26:49.336852 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.336831    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-77.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 16:26:49.336915 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.336833    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 16:26:49.337083 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.337068    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-77.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 16:26:49.339029 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.339018    2565 server.go:1268] "Started kubelet"
Sep 15 16:26:49.339168 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.339082    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 16:26:49.339216 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.339122    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 16:26:49.339216 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.339201    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 16:26:49.339788 ip-10-0-134-77 systemd[1]: Started Kubernetes Kubelet.
Sep 15 16:26:49.345183 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.345159    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 16:26:49.346971 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.346922    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 16:26:49.347049 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.347030    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6zl59"
Sep 15 16:26:49.348169 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.348154    2565 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 16:26:49.350424 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.350405    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6zl59"
Sep 15 16:26:49.354206 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.354188    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 16:26:49.355068 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.355049    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 16:26:49.355204 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.355188    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:49.355842 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.355825    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 16:26:49.355971 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.355960    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 16:26:49.356065 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.355824    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 16:26:49.356119 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.356109    2565 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 16:26:49.356119 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.356116    2565 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 16:26:49.357294 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.357215    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 16:26:49.357294 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.357231    2565 factory.go:55] Registering systemd factory
Sep 15 16:26:49.357294 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.357239    2565 factory.go:223] Registration of the systemd container factory successfully
Sep 15 16:26:49.357534 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.357516    2565 factory.go:153] Registering CRI-O factory
Sep 15 16:26:49.357607 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.357538    2565 factory.go:223] Registration of the crio container factory successfully
Sep 15 16:26:49.357607 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.357555    2565 factory.go:103] Registering Raw factory
Sep 15 16:26:49.357607 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.357568    2565 manager.go:1196] Started watching for new ooms in manager
Sep 15 16:26:49.357771 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.357627    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:49.357974 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.357962    2565 manager.go:349] Starting recovery of all containers
Sep 15 16:26:49.360425 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.360406    2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-77.ec2.internal\" not found" node="ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.368853 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.368839    2565 manager.go:354] Recovery completed
Sep 15 16:26:49.370200 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.370015    2565 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 15 16:26:49.373148 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.373132    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:49.374955 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.374939    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:49.375023 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.374979    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:49.375023 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.374994    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:49.375668 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.375649    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 16:26:49.375668 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.375665    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 16:26:49.375823 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.375673    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:49.375823 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.375682    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 16:26:49.376508 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.376489    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:49.376601 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.376523    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:49.376601 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.376537    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:49.376833 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.376820    2565 policy_none.go:49] "None policy: Start"
Sep 15 16:26:49.376890 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.376838    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 16:26:49.376890 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.376850    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 16:26:49.377865 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.377854    2565 policy_none.go:47] "Start"
Sep 15 16:26:49.403314 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.403296    2565 manager.go:337] "Starting Device Plugin manager"
Sep 15 16:26:49.415180 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.403341    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 16:26:49.415180 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.403351    2565 server.go:87] "Starting device plugin registration server"
Sep 15 16:26:49.415180 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.403622    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 16:26:49.415180 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.403635    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 16:26:49.415180 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.403737    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 16:26:49.415180 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.403838    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 16:26:49.415180 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.403847    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 16:26:49.415180 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.404452    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 16:26:49.415180 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.404518    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:49.442088 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.442057    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 16:26:49.443220 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.443202    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 16:26:49.443293 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.443254    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 16:26:49.443293 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.443284    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 16:26:49.443383 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.443329    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 16:26:49.445686 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.445666    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:49.504160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.504115    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:49.505140 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.505118    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:49.505248 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.505152    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:49.505248 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.505169    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:49.505248 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.505193    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.514560 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.514541    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.514640 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.514564    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-77.ec2.internal\": node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:49.533164 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.533142    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:49.543906 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.543881    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-77.ec2.internal"]
Sep 15 16:26:49.543992 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.543966    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:49.545264 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.545248    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:49.545347 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.545278    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:49.545347 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.545289    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:49.546305 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.546291    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:49.546434 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.546419    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.546508 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.546450    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:49.547041 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.547025    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:49.547041 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.547038    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:49.547162 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.547057    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:49.547162 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.547058    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:49.547162 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.547068    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:49.547162 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.547074    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:49.548022 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.548008    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.548079 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.548034    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:49.548878 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.548862    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:49.548973 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.548887    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:49.548973 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.548900    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:49.557257 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.557234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a675b393087d5d8948209c59db07488b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal\" (UID: \"a675b393087d5d8948209c59db07488b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.557318 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.557265    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a675b393087d5d8948209c59db07488b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal\" (UID: \"a675b393087d5d8948209c59db07488b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.557318 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.557282    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce9c1ce3a3fab75c3e888d6b8fff341b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-77.ec2.internal\" (UID: \"ce9c1ce3a3fab75c3e888d6b8fff341b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.577339 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.577319    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-77.ec2.internal\" not found" node="ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.581842 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.581829    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-77.ec2.internal\" not found" node="ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.634258 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.634224    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:49.658013 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.657990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a675b393087d5d8948209c59db07488b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal\" (UID: \"a675b393087d5d8948209c59db07488b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.658099 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.658017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a675b393087d5d8948209c59db07488b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal\" (UID: \"a675b393087d5d8948209c59db07488b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.658099 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.658041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce9c1ce3a3fab75c3e888d6b8fff341b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-77.ec2.internal\" (UID: \"ce9c1ce3a3fab75c3e888d6b8fff341b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.658163 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.658096    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce9c1ce3a3fab75c3e888d6b8fff341b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-77.ec2.internal\" (UID: \"ce9c1ce3a3fab75c3e888d6b8fff341b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.658163 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.658105    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a675b393087d5d8948209c59db07488b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal\" (UID: \"a675b393087d5d8948209c59db07488b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.658163 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.658111    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a675b393087d5d8948209c59db07488b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal\" (UID: \"a675b393087d5d8948209c59db07488b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.735366 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.735344    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:49.836221 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.836156    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:49.879693 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.879662    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.884573 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:49.884552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:49.937103 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:49.936950    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:50.037472 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:50.037437    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:50.138037 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:50.137972    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:50.238465 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:50.238436    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:50.288966 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:50.288938    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 16:26:50.289395 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:50.289083    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 16:26:50.289395 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:50.289104    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 16:26:50.339027 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:50.339002    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:50.352496 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:50.352459    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 16:21:49 +0000 UTC" deadline="2028-02-17 21:50:44.769135026 +0000 UTC"
Sep 15 16:26:50.352496 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:50.352491    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12485h23m54.416647734s"
Sep 15 16:26:50.439470 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:50.439425    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:50.458112 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:50.458081    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:50.540271 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:50.540231    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:50.552995 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:50.552967    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9c1ce3a3fab75c3e888d6b8fff341b.slice/crio-5b984ab73947b712a00465851756f05962a9295108199388d81792367ca9b2a7 WatchSource:0}: Error finding container 5b984ab73947b712a00465851756f05962a9295108199388d81792367ca9b2a7: Status 404 returned error can't find the container with id 5b984ab73947b712a00465851756f05962a9295108199388d81792367ca9b2a7
Sep 15 16:26:50.553489 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:50.553460    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda675b393087d5d8948209c59db07488b.slice/crio-d43c17ae8e7f9216fe0be2bf2dd480675ba648ac89590e5a03fa1280b39cabe0 WatchSource:0}: Error finding container d43c17ae8e7f9216fe0be2bf2dd480675ba648ac89590e5a03fa1280b39cabe0: Status 404 returned error can't find the container with id d43c17ae8e7f9216fe0be2bf2dd480675ba648ac89590e5a03fa1280b39cabe0
Sep 15 16:26:50.560107 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:50.560088    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:26:50.642028 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:50.641975    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:50.743252 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:50.743155    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-77.ec2.internal\" not found"
Sep 15 16:26:50.798496 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:50.798471    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:50.855800 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:50.855758    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:50.867966 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:50.867944    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 16:26:50.869052 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:50.869037    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-77.ec2.internal"
Sep 15 16:26:50.881870 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:50.881847    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 16:26:51.188860 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.188618    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:51.328769 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.328732    2565 apiserver.go:52] "Watching apiserver"
Sep 15 16:26:51.340449 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.340419    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 16:26:51.342138 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.342099    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-9bwpd","kube-system/global-pull-secret-syncer-jvr5t","kube-system/konnectivity-agent-fjwgp","kube-system/kube-apiserver-proxy-ip-10-0-134-77.ec2.internal","openshift-cluster-node-tuning-operator/tuned-mnwxq","openshift-dns/node-resolver-mj466","openshift-image-registry/node-ca-7np85","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal","openshift-multus/multus-additional-cni-plugins-cxxfh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267","openshift-multus/multus-txbkb","openshift-multus/network-metrics-daemon-nvl8x","openshift-network-diagnostics/network-check-target-vjsq5","openshift-network-operator/iptables-alerter-kws86"]
Sep 15 16:26:51.345226 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.345202    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.348132 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.348106    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.350088 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.349838    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 16:26:51.350088 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.349894    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mgtgz\""
Sep 15 16:26:51.350088 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.349986    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 16:26:51.350308 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.350096    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 16:26:51.351195 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.350890    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fjwgp"
Sep 15 16:26:51.351195 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.350955    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 16:26:51.351195 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.350981    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 16:26:51.351195 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.351007    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-shsr5\""
Sep 15 16:26:51.351893 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.351873    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 16:26:51.352377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.352134    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 16:26:51.352377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.352202    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 16:26:51.352579 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.352408    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 16:26:51.353380 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.353359    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-trxkn\""
Sep 15 16:26:51.353517 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.353499    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 16:26:51.353667 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.353649    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:51.353759 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:51.353733    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:26:51.354155 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.354133    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 16:26:51.354856 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.354835    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 16:26:51.356514 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.356492    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7np85"
Sep 15 16:26:51.358875 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.358854    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 16:26:51.359149 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.359132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.359237 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.359223    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 16:26:51.359307 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.359283    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 16:26:51.359987 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.359962    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hdq26\""
Sep 15 16:26:51.361882 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.361852    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 16:26:51.361882 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.361873    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 16:26:51.362039 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.361859    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d9cqm\""
Sep 15 16:26:51.362039 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.361855    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 16:26:51.362228 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.362197    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 16:26:51.362340 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.362327    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.364825 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.364751    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 16:26:51.364825 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.364767    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-trdvs\""
Sep 15 16:26:51.364982 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.364867    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 16:26:51.365729 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.365361    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.368036 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.368012    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:51.368373 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.368354    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4jxq9\""
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.369386    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:51.369415    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.369473    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.369700    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac156fa7-e35f-4265-baa3-be8f567a8190-ovn-node-metrics-cert\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.369885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-run-netns\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.369924    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-systemd-units\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.370278    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.369954    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-run-ovn\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.370554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-node-log\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.370593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15e6809f-8f4b-4eeb-ba75-9ee8226ad2da-agent-certs\") pod \"konnectivity-agent-fjwgp\" (UID: \"15e6809f-8f4b-4eeb-ba75-9ee8226ad2da\") " pod="kube-system/konnectivity-agent-fjwgp"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.370653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a34bc31-c8e6-4214-9762-366d6fe23190-serviceca\") pod \"node-ca-7np85\" (UID: \"5a34bc31-c8e6-4214-9762-366d6fe23190\") " pod="openshift-image-registry/node-ca-7np85"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.370690    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-var-lib-cni-multus\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.370740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-multus-conf-dir\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.370771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-run-multus-certs\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.370800    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.370830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-socket-dir\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.370866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.370862    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-slash\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.370892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-etc-openvswitch\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.370943    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-run-ovn-kubernetes\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.370970    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac156fa7-e35f-4265-baa3-be8f567a8190-ovnkube-script-lib\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs\") pod \"network-metrics-daemon-nvl8x\" (UID: \"9a34ad6e-33f8-4f96-9aca-98998f3bcd62\") " pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-run-netns\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac156fa7-e35f-4265-baa3-be8f567a8190-ovnkube-config\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wldmt\" (UniqueName: \"kubernetes.io/projected/ac156fa7-e35f-4265-baa3-be8f567a8190-kube-api-access-wldmt\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-cnibin\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371188    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-run-k8s-cni-cncf-io\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371216    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-hostroot\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371245    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f7d621a-2df5-4eb9-b76c-a8e244869670-multus-daemon-config\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371275    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-cni-bin\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a34bc31-c8e6-4214-9762-366d6fe23190-host\") pod \"node-ca-7np85\" (UID: \"5a34bc31-c8e6-4214-9762-366d6fe23190\") " pod="openshift-image-registry/node-ca-7np85"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371327    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-system-cni-dir\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f7d621a-2df5-4eb9-b76c-a8e244869670-cni-binary-copy\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-etc-kubernetes\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.371756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371433    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-device-dir\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371463    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-sys-fs\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-var-lib-openvswitch\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371518    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7km4\" (UniqueName: \"kubernetes.io/projected/5a34bc31-c8e6-4214-9762-366d6fe23190-kube-api-access-p7km4\") pod \"node-ca-7np85\" (UID: \"5a34bc31-c8e6-4214-9762-366d6fe23190\") " pod="openshift-image-registry/node-ca-7np85"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-multus-cni-dir\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371580    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-multus-socket-dir-parent\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371604    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-var-lib-cni-bin\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371635    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh5dc\" (UniqueName: \"kubernetes.io/projected/2f7d621a-2df5-4eb9-b76c-a8e244869670-kube-api-access-kh5dc\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-etc-selinux\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371703    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15e6809f-8f4b-4eeb-ba75-9ee8226ad2da-konnectivity-ca\") pod \"konnectivity-agent-fjwgp\" (UID: \"15e6809f-8f4b-4eeb-ba75-9ee8226ad2da\") " pod="kube-system/konnectivity-agent-fjwgp"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371764    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-var-lib-kubelet\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-run-openvswitch\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac156fa7-e35f-4265-baa3-be8f567a8190-env-overrides\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjscb\" (UniqueName: \"kubernetes.io/projected/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-kube-api-access-zjscb\") pod \"network-metrics-daemon-nvl8x\" (UID: \"9a34ad6e-33f8-4f96-9aca-98998f3bcd62\") " pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-registration-dir\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.372632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.371971    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjghf\" (UniqueName: \"kubernetes.io/projected/21dbb64e-2429-48df-bfe2-a7cb863cec30-kube-api-access-fjghf\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.373461 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.372002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-kubelet\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.373461 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.372026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-run-systemd\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.373461 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.372056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-log-socket\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.373461 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.372089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-cni-netd\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.374280 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.374231    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:26:51.374371 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:51.374336    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:26:51.377260 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.376902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mj466"
Sep 15 16:26:51.379338 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.379313    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 16:26:51.379441 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.379361    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-79pft\""
Sep 15 16:26:51.379624 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.379603    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 16:26:51.380164 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.380143    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kws86"
Sep 15 16:26:51.382488 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.382425    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 16:26:51.382718 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.382571    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 16:26:51.382868 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.382728    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xpm5p\""
Sep 15 16:26:51.383534 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.383381    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 16:26:51.411314 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.411291    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ch7rt"
Sep 15 16:26:51.422092 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.422071    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ch7rt"
Sep 15 16:26:51.449932 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.449844    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal" event={"ID":"a675b393087d5d8948209c59db07488b","Type":"ContainerStarted","Data":"d43c17ae8e7f9216fe0be2bf2dd480675ba648ac89590e5a03fa1280b39cabe0"}
Sep 15 16:26:51.451813 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.451782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-77.ec2.internal" event={"ID":"ce9c1ce3a3fab75c3e888d6b8fff341b","Type":"ContainerStarted","Data":"5b984ab73947b712a00465851756f05962a9295108199388d81792367ca9b2a7"}
Sep 15 16:26:51.458083 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.458061    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 16:26:51.472255 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjscb\" (UniqueName: \"kubernetes.io/projected/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-kube-api-access-zjscb\") pod \"network-metrics-daemon-nvl8x\" (UID: \"9a34ad6e-33f8-4f96-9aca-98998f3bcd62\") " pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:51.472372 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472267    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:51.472372 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-registration-dir\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.472372 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjghf\" (UniqueName: \"kubernetes.io/projected/21dbb64e-2429-48df-bfe2-a7cb863cec30-kube-api-access-fjghf\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.472372 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-kubelet\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.472592 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472400    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-kubelet\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.472592 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472401    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-run-systemd\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.472592 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-log-socket\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.472592 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472449    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-run-systemd\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.472592 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472467    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-cni-netd\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.472949 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-run-netns\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.472949 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472634    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-systemd-units\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.472949 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-node-log\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.472949 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472692    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-var-lib-cni-multus\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.472949 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472734    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-multus-conf-dir\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.472949 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-run-netns\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.472949 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-var-lib-cni-multus\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.472949 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-multus-conf-dir\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.472949 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-systemd-units\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.472949 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472880    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-cni-netd\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.472949 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472932    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-log-socket\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.473440 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7764505-fade-42cf-9a24-191b27545755-tmp-dir\") pod \"node-resolver-mj466\" (UID: \"b7764505-fade-42cf-9a24-191b27545755\") " pod="openshift-dns/node-resolver-mj466"
Sep 15 16:26:51.473440 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.472990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79a248eb-edf5-486b-b941-217db1abc9ea-iptables-alerter-script\") pod \"iptables-alerter-kws86\" (UID: \"79a248eb-edf5-486b-b941-217db1abc9ea\") " pod="openshift-network-operator/iptables-alerter-kws86"
Sep 15 16:26:51.473440 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473028    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-kubernetes\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.473440 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473047    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-registration-dir\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.473440 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-run\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.473440 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473084    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-node-log\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.473440 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-slash\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.473440 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-etc-openvswitch\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.473440 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-slash\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.473440 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-run-ovn-kubernetes\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.473440 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473260    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-run-ovn-kubernetes\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.473440 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac156fa7-e35f-4265-baa3-be8f567a8190-ovnkube-script-lib\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.473440 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs\") pod \"network-metrics-daemon-nvl8x\" (UID: \"9a34ad6e-33f8-4f96-9aca-98998f3bcd62\") " pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:51.474160 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:51.473466    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:51.474160 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:51.473611    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs podName:9a34ad6e-33f8-4f96-9aca-98998f3bcd62 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:51.973572808 +0000 UTC m=+3.017958520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs") pod "network-metrics-daemon-nvl8x" (UID: "9a34ad6e-33f8-4f96-9aca-98998f3bcd62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:51.474160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/150d8a61-545c-417a-b159-f2e84473250a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.474160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32b7532a-7fdd-4341-881f-2715c75f2b88-tmp\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.474160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-etc-kubernetes\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.474160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-run-netns\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.474160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac156fa7-e35f-4265-baa3-be8f567a8190-ovnkube-config\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.474160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.473964    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wldmt\" (UniqueName: \"kubernetes.io/projected/ac156fa7-e35f-4265-baa3-be8f567a8190-kube-api-access-wldmt\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.474160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb4nr\" (UniqueName: \"kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr\") pod \"network-check-target-vjsq5\" (UID: \"33198472-1392-416c-a639-0ba97f72fd7f\") " pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:26:51.474160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-577xv\" (UniqueName: \"kubernetes.io/projected/79a248eb-edf5-486b-b941-217db1abc9ea-kube-api-access-577xv\") pod \"iptables-alerter-kws86\" (UID: \"79a248eb-edf5-486b-b941-217db1abc9ea\") " pod="openshift-network-operator/iptables-alerter-kws86"
Sep 15 16:26:51.474160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-lib-modules\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.474160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-var-lib-kubelet\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474174    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-cni-bin\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474206    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-tuned\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-device-dir\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474274    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-multus-cni-dir\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac156fa7-e35f-4265-baa3-be8f567a8190-ovnkube-script-lib\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474394    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-etc-kubernetes\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-multus-socket-dir-parent\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474435    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-run-netns\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-etc-selinux\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474471    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/150d8a61-545c-417a-b159-f2e84473250a-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-systemd\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zpf6\" (UniqueName: \"kubernetes.io/projected/32b7532a-7fdd-4341-881f-2715c75f2b88-kube-api-access-7zpf6\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac156fa7-e35f-4265-baa3-be8f567a8190-env-overrides\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/150d8a61-545c-417a-b159-f2e84473250a-cnibin\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.474761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-host\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.474978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac156fa7-e35f-4265-baa3-be8f567a8190-ovnkube-config\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac156fa7-e35f-4265-baa3-be8f567a8190-ovn-node-metrics-cert\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-run-multus-certs\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/150d8a61-545c-417a-b159-f2e84473250a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475153    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-device-dir\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-etc-openvswitch\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/22495e12-7c88-4178-9d83-076062a2c424-dbus\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-run-ovn\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-multus-socket-dir-parent\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475252    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15e6809f-8f4b-4eeb-ba75-9ee8226ad2da-agent-certs\") pod \"konnectivity-agent-fjwgp\" (UID: \"15e6809f-8f4b-4eeb-ba75-9ee8226ad2da\") " pod="kube-system/konnectivity-agent-fjwgp"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a34bc31-c8e6-4214-9762-366d6fe23190-serviceca\") pod \"node-ca-7np85\" (UID: \"5a34bc31-c8e6-4214-9762-366d6fe23190\") " pod="openshift-image-registry/node-ca-7np85"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475303    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-sysctl-conf\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/22495e12-7c88-4178-9d83-076062a2c424-kubelet-config\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-etc-selinux\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh5dc\" (UniqueName: \"kubernetes.io/projected/2f7d621a-2df5-4eb9-b76c-a8e244869670-kube-api-access-kh5dc\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475411    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-multus-cni-dir\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.475538 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-run-ovn\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475795    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac156fa7-e35f-4265-baa3-be8f567a8190-env-overrides\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475857    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-cni-bin\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475879    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a34bc31-c8e6-4214-9762-366d6fe23190-serviceca\") pod \"node-ca-7np85\" (UID: \"5a34bc31-c8e6-4214-9762-366d6fe23190\") " pod="openshift-image-registry/node-ca-7np85"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.475994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-run-multus-certs\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-socket-dir\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-cnibin\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476116    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-run-k8s-cni-cncf-io\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476196    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7764505-fade-42cf-9a24-191b27545755-hosts-file\") pod \"node-resolver-mj466\" (UID: \"b7764505-fade-42cf-9a24-191b27545755\") " pod="openshift-dns/node-resolver-mj466"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476210    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-cnibin\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-socket-dir\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79a248eb-edf5-486b-b941-217db1abc9ea-host-slash\") pod \"iptables-alerter-kws86\" (UID: \"79a248eb-edf5-486b-b941-217db1abc9ea\") " pod="openshift-network-operator/iptables-alerter-kws86"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476255    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-run-k8s-cni-cncf-io\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476294    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-hostroot\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476328    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-hostroot\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f7d621a-2df5-4eb9-b76c-a8e244869670-multus-daemon-config\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.476397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476350    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476386    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb24t\" (UniqueName: \"kubernetes.io/projected/150d8a61-545c-417a-b159-f2e84473250a-kube-api-access-bb24t\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-modprobe-d\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476457    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-sysctl-d\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476563    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-sys\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a34bc31-c8e6-4214-9762-366d6fe23190-host\") pod \"node-ca-7np85\" (UID: \"5a34bc31-c8e6-4214-9762-366d6fe23190\") " pod="openshift-image-registry/node-ca-7np85"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476649    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-system-cni-dir\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476692    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a34bc31-c8e6-4214-9762-366d6fe23190-host\") pod \"node-ca-7np85\" (UID: \"5a34bc31-c8e6-4214-9762-366d6fe23190\") " pod="openshift-image-registry/node-ca-7np85"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f7d621a-2df5-4eb9-b76c-a8e244869670-cni-binary-copy\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-system-cni-dir\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/150d8a61-545c-417a-b159-f2e84473250a-system-cni-dir\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-sys-fs\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.476954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-var-lib-openvswitch\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7km4\" (UniqueName: \"kubernetes.io/projected/5a34bc31-c8e6-4214-9762-366d6fe23190-kube-api-access-p7km4\") pod \"node-ca-7np85\" (UID: \"5a34bc31-c8e6-4214-9762-366d6fe23190\") " pod="openshift-image-registry/node-ca-7np85"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477009    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/21dbb64e-2429-48df-bfe2-a7cb863cec30-sys-fs\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-var-lib-cni-bin\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477053    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-var-lib-openvswitch\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.477285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477143    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/150d8a61-545c-417a-b159-f2e84473250a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.478125 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477191    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-var-lib-cni-bin\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.478125 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477248    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26v5b\" (UniqueName: \"kubernetes.io/projected/b7764505-fade-42cf-9a24-191b27545755-kube-api-access-26v5b\") pod \"node-resolver-mj466\" (UID: \"b7764505-fade-42cf-9a24-191b27545755\") " pod="openshift-dns/node-resolver-mj466"
Sep 15 16:26:51.478125 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477277    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f7d621a-2df5-4eb9-b76c-a8e244869670-multus-daemon-config\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.478125 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477303    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-sysconfig\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.478125 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.478125 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15e6809f-8f4b-4eeb-ba75-9ee8226ad2da-konnectivity-ca\") pod \"konnectivity-agent-fjwgp\" (UID: \"15e6809f-8f4b-4eeb-ba75-9ee8226ad2da\") " pod="kube-system/konnectivity-agent-fjwgp"
Sep 15 16:26:51.478125 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477376    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f7d621a-2df5-4eb9-b76c-a8e244869670-cni-binary-copy\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.478125 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.478125 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-var-lib-kubelet\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.478125 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477502    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f7d621a-2df5-4eb9-b76c-a8e244869670-host-var-lib-kubelet\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.478125 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-run-openvswitch\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.478125 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477605    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac156fa7-e35f-4265-baa3-be8f567a8190-run-openvswitch\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.478125 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.477964    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15e6809f-8f4b-4eeb-ba75-9ee8226ad2da-konnectivity-ca\") pod \"konnectivity-agent-fjwgp\" (UID: \"15e6809f-8f4b-4eeb-ba75-9ee8226ad2da\") " pod="kube-system/konnectivity-agent-fjwgp"
Sep 15 16:26:51.480990 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.480943    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac156fa7-e35f-4265-baa3-be8f567a8190-ovn-node-metrics-cert\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.481186 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.481138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15e6809f-8f4b-4eeb-ba75-9ee8226ad2da-agent-certs\") pod \"konnectivity-agent-fjwgp\" (UID: \"15e6809f-8f4b-4eeb-ba75-9ee8226ad2da\") " pod="kube-system/konnectivity-agent-fjwgp"
Sep 15 16:26:51.485382 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.485348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjscb\" (UniqueName: \"kubernetes.io/projected/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-kube-api-access-zjscb\") pod \"network-metrics-daemon-nvl8x\" (UID: \"9a34ad6e-33f8-4f96-9aca-98998f3bcd62\") " pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:51.487493 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.487353    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh5dc\" (UniqueName: \"kubernetes.io/projected/2f7d621a-2df5-4eb9-b76c-a8e244869670-kube-api-access-kh5dc\") pod \"multus-txbkb\" (UID: \"2f7d621a-2df5-4eb9-b76c-a8e244869670\") " pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.488489 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.488450    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wldmt\" (UniqueName: \"kubernetes.io/projected/ac156fa7-e35f-4265-baa3-be8f567a8190-kube-api-access-wldmt\") pod \"ovnkube-node-9bwpd\" (UID: \"ac156fa7-e35f-4265-baa3-be8f567a8190\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.489169 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.489143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjghf\" (UniqueName: \"kubernetes.io/projected/21dbb64e-2429-48df-bfe2-a7cb863cec30-kube-api-access-fjghf\") pod \"aws-ebs-csi-driver-node-4p267\" (UID: \"21dbb64e-2429-48df-bfe2-a7cb863cec30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.490610 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.490569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7km4\" (UniqueName: \"kubernetes.io/projected/5a34bc31-c8e6-4214-9762-366d6fe23190-kube-api-access-p7km4\") pod \"node-ca-7np85\" (UID: \"5a34bc31-c8e6-4214-9762-366d6fe23190\") " pod="openshift-image-registry/node-ca-7np85"
Sep 15 16:26:51.578104 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7764505-fade-42cf-9a24-191b27545755-hosts-file\") pod \"node-resolver-mj466\" (UID: \"b7764505-fade-42cf-9a24-191b27545755\") " pod="openshift-dns/node-resolver-mj466"
Sep 15 16:26:51.578272 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578109    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79a248eb-edf5-486b-b941-217db1abc9ea-host-slash\") pod \"iptables-alerter-kws86\" (UID: \"79a248eb-edf5-486b-b941-217db1abc9ea\") " pod="openshift-network-operator/iptables-alerter-kws86"
Sep 15 16:26:51.578272 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb24t\" (UniqueName: \"kubernetes.io/projected/150d8a61-545c-417a-b159-f2e84473250a-kube-api-access-bb24t\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.578272 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578201    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7764505-fade-42cf-9a24-191b27545755-hosts-file\") pod \"node-resolver-mj466\" (UID: \"b7764505-fade-42cf-9a24-191b27545755\") " pod="openshift-dns/node-resolver-mj466"
Sep 15 16:26:51.578272 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79a248eb-edf5-486b-b941-217db1abc9ea-host-slash\") pod \"iptables-alerter-kws86\" (UID: \"79a248eb-edf5-486b-b941-217db1abc9ea\") " pod="openshift-network-operator/iptables-alerter-kws86"
Sep 15 16:26:51.578500 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-modprobe-d\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.578500 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578366    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-sysctl-d\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.578500 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-sys\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.578500 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-sys\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.578500 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-modprobe-d\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.578715 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578498    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-sysctl-d\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.578715 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/150d8a61-545c-417a-b159-f2e84473250a-system-cni-dir\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.578715 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/150d8a61-545c-417a-b159-f2e84473250a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.578715 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/150d8a61-545c-417a-b159-f2e84473250a-system-cni-dir\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.578715 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26v5b\" (UniqueName: \"kubernetes.io/projected/b7764505-fade-42cf-9a24-191b27545755-kube-api-access-26v5b\") pod \"node-resolver-mj466\" (UID: \"b7764505-fade-42cf-9a24-191b27545755\") " pod="openshift-dns/node-resolver-mj466"
Sep 15 16:26:51.578715 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-sysconfig\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.578715 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578649    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:51.578715 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7764505-fade-42cf-9a24-191b27545755-tmp-dir\") pod \"node-resolver-mj466\" (UID: \"b7764505-fade-42cf-9a24-191b27545755\") " pod="openshift-dns/node-resolver-mj466"
Sep 15 16:26:51.578715 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578698    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-sysconfig\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.578715 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79a248eb-edf5-486b-b941-217db1abc9ea-iptables-alerter-script\") pod \"iptables-alerter-kws86\" (UID: \"79a248eb-edf5-486b-b941-217db1abc9ea\") " pod="openshift-network-operator/iptables-alerter-kws86"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578764    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-kubernetes\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-kubernetes\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:51.578838    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:51.578898    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret podName:22495e12-7c88-4178-9d83-076062a2c424 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:52.078878554 +0000 UTC m=+3.123264283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret") pod "global-pull-secret-syncer-jvr5t" (UID: "22495e12-7c88-4178-9d83-076062a2c424") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-run\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578968    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/150d8a61-545c-417a-b159-f2e84473250a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.578994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32b7532a-7fdd-4341-881f-2715c75f2b88-tmp\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4nr\" (UniqueName: \"kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr\") pod \"network-check-target-vjsq5\" (UID: \"33198472-1392-416c-a639-0ba97f72fd7f\") " pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-577xv\" (UniqueName: \"kubernetes.io/projected/79a248eb-edf5-486b-b941-217db1abc9ea-kube-api-access-577xv\") pod \"iptables-alerter-kws86\" (UID: \"79a248eb-edf5-486b-b941-217db1abc9ea\") " pod="openshift-network-operator/iptables-alerter-kws86"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579072    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7764505-fade-42cf-9a24-191b27545755-tmp-dir\") pod \"node-resolver-mj466\" (UID: \"b7764505-fade-42cf-9a24-191b27545755\") " pod="openshift-dns/node-resolver-mj466"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-lib-modules\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-var-lib-kubelet\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-tuned\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/150d8a61-545c-417a-b159-f2e84473250a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579184    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/150d8a61-545c-417a-b159-f2e84473250a-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.579239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-systemd\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579251    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/150d8a61-545c-417a-b159-f2e84473250a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpf6\" (UniqueName: \"kubernetes.io/projected/32b7532a-7fdd-4341-881f-2715c75f2b88-kube-api-access-7zpf6\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/150d8a61-545c-417a-b159-f2e84473250a-cnibin\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579324    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-host\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579329    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-run\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/150d8a61-545c-417a-b159-f2e84473250a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579356    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-systemd\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/150d8a61-545c-417a-b159-f2e84473250a-cnibin\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/22495e12-7c88-4178-9d83-076062a2c424-dbus\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579400    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79a248eb-edf5-486b-b941-217db1abc9ea-iptables-alerter-script\") pod \"iptables-alerter-kws86\" (UID: \"79a248eb-edf5-486b-b941-217db1abc9ea\") " pod="openshift-network-operator/iptables-alerter-kws86"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579444    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/22495e12-7c88-4178-9d83-076062a2c424-dbus\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579479    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-host\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-sysctl-conf\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/22495e12-7c88-4178-9d83-076062a2c424-kubelet-config\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579525    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-lib-modules\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579623    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/22495e12-7c88-4178-9d83-076062a2c424-kubelet-config\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:51.580015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/150d8a61-545c-417a-b159-f2e84473250a-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.580639 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579756    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-sysctl-conf\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.580639 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579771    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32b7532a-7fdd-4341-881f-2715c75f2b88-var-lib-kubelet\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.580639 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.579957    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/150d8a61-545c-417a-b159-f2e84473250a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.581498 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.581471    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32b7532a-7fdd-4341-881f-2715c75f2b88-etc-tuned\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.581607 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.581516    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32b7532a-7fdd-4341-881f-2715c75f2b88-tmp\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.585865 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:51.585810    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:51.585865 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:51.585839    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:51.585865 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:51.585853    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sb4nr for pod openshift-network-diagnostics/network-check-target-vjsq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:51.586139 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:51.585919    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr podName:33198472-1392-416c-a639-0ba97f72fd7f nodeName:}" failed. No retries permitted until 2026-09-15 16:26:52.085902221 +0000 UTC m=+3.130287936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sb4nr" (UniqueName: "kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr") pod "network-check-target-vjsq5" (UID: "33198472-1392-416c-a639-0ba97f72fd7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:51.586784 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.586730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26v5b\" (UniqueName: \"kubernetes.io/projected/b7764505-fade-42cf-9a24-191b27545755-kube-api-access-26v5b\") pod \"node-resolver-mj466\" (UID: \"b7764505-fade-42cf-9a24-191b27545755\") " pod="openshift-dns/node-resolver-mj466"
Sep 15 16:26:51.587304 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.587278    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb24t\" (UniqueName: \"kubernetes.io/projected/150d8a61-545c-417a-b159-f2e84473250a-kube-api-access-bb24t\") pod \"multus-additional-cni-plugins-cxxfh\" (UID: \"150d8a61-545c-417a-b159-f2e84473250a\") " pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.587726 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.587703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-577xv\" (UniqueName: \"kubernetes.io/projected/79a248eb-edf5-486b-b941-217db1abc9ea-kube-api-access-577xv\") pod \"iptables-alerter-kws86\" (UID: \"79a248eb-edf5-486b-b941-217db1abc9ea\") " pod="openshift-network-operator/iptables-alerter-kws86"
Sep 15 16:26:51.587977 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.587956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zpf6\" (UniqueName: \"kubernetes.io/projected/32b7532a-7fdd-4341-881f-2715c75f2b88-kube-api-access-7zpf6\") pod \"tuned-mnwxq\" (UID: \"32b7532a-7fdd-4341-881f-2715c75f2b88\") " pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.662261 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.662224    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267"
Sep 15 16:26:51.671698 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.671672    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:26:51.683908 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.683884    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fjwgp"
Sep 15 16:26:51.689923 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.689897    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7np85"
Sep 15 16:26:51.698945 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.698924    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-txbkb"
Sep 15 16:26:51.706770 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.706753    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cxxfh"
Sep 15 16:26:51.715659 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.715636    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mnwxq"
Sep 15 16:26:51.724650 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.724631    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mj466"
Sep 15 16:26:51.730436 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.730411    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kws86"
Sep 15 16:26:51.793123 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.793098    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:51.983036 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:51.982960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs\") pod \"network-metrics-daemon-nvl8x\" (UID: \"9a34ad6e-33f8-4f96-9aca-98998f3bcd62\") " pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:51.983174 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:51.983153    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:51.983239 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:51.983226    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs podName:9a34ad6e-33f8-4f96-9aca-98998f3bcd62 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:52.983209762 +0000 UTC m=+4.027595478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs") pod "network-metrics-daemon-nvl8x" (UID: "9a34ad6e-33f8-4f96-9aca-98998f3bcd62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:52.084026 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:52.083982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:52.084204 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:52.084141    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:52.084259 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:52.084225    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret podName:22495e12-7c88-4178-9d83-076062a2c424 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:53.084202877 +0000 UTC m=+4.128588599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret") pod "global-pull-secret-syncer-jvr5t" (UID: "22495e12-7c88-4178-9d83-076062a2c424") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:52.184621 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:52.184587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4nr\" (UniqueName: \"kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr\") pod \"network-check-target-vjsq5\" (UID: \"33198472-1392-416c-a639-0ba97f72fd7f\") " pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:26:52.184815 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:52.184728    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:52.184815 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:52.184760    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:52.184815 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:52.184770    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sb4nr for pod openshift-network-diagnostics/network-check-target-vjsq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:52.184947 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:52.184837    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr podName:33198472-1392-416c-a639-0ba97f72fd7f nodeName:}" failed. No retries permitted until 2026-09-15 16:26:53.184811655 +0000 UTC m=+4.229197384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sb4nr" (UniqueName: "kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr") pod "network-check-target-vjsq5" (UID: "33198472-1392-416c-a639-0ba97f72fd7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:52.426285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:52.426198    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 16:21:51 +0000 UTC" deadline="2028-03-18 14:28:49.750073393 +0000 UTC"
Sep 15 16:26:52.426285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:52.426231    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13198h1m57.323845792s"
Sep 15 16:26:52.722673 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:52.722638    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21dbb64e_2429_48df_bfe2_a7cb863cec30.slice/crio-6d6717c62afd94b1fbac3c849acac3df77f81bc66c552ebfe67f2d61d4c5b9f1 WatchSource:0}: Error finding container 6d6717c62afd94b1fbac3c849acac3df77f81bc66c552ebfe67f2d61d4c5b9f1: Status 404 returned error can't find the container with id 6d6717c62afd94b1fbac3c849acac3df77f81bc66c552ebfe67f2d61d4c5b9f1
Sep 15 16:26:52.723891 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:52.723844    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b7532a_7fdd_4341_881f_2715c75f2b88.slice/crio-bec2332aa20682534942af711ea32bdd205941625c159dc626ba7b8f6f0a39dc WatchSource:0}: Error finding container bec2332aa20682534942af711ea32bdd205941625c159dc626ba7b8f6f0a39dc: Status 404 returned error can't find the container with id bec2332aa20682534942af711ea32bdd205941625c159dc626ba7b8f6f0a39dc
Sep 15 16:26:52.728006 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:52.727864    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a34bc31_c8e6_4214_9762_366d6fe23190.slice/crio-18bb30af7e32053dbe4e1b7b3e11b1a3f4c709490a4a7319f7d48bc52e7e4c9c WatchSource:0}: Error finding container 18bb30af7e32053dbe4e1b7b3e11b1a3f4c709490a4a7319f7d48bc52e7e4c9c: Status 404 returned error can't find the container with id 18bb30af7e32053dbe4e1b7b3e11b1a3f4c709490a4a7319f7d48bc52e7e4c9c
Sep 15 16:26:52.729127 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:52.729101    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a248eb_edf5_486b_b941_217db1abc9ea.slice/crio-6e0aa40884d372e8e37b8177da5b16affcc05f177a69b588e3c544e5ea633a9a WatchSource:0}: Error finding container 6e0aa40884d372e8e37b8177da5b16affcc05f177a69b588e3c544e5ea633a9a: Status 404 returned error can't find the container with id 6e0aa40884d372e8e37b8177da5b16affcc05f177a69b588e3c544e5ea633a9a
Sep 15 16:26:52.729773 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:52.729655    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7d621a_2df5_4eb9_b76c_a8e244869670.slice/crio-8593cbc7eed057cfc2837e8024968004c527097b9923f4b9cfb82f044e2b0dcc WatchSource:0}: Error finding container 8593cbc7eed057cfc2837e8024968004c527097b9923f4b9cfb82f044e2b0dcc: Status 404 returned error can't find the container with id 8593cbc7eed057cfc2837e8024968004c527097b9923f4b9cfb82f044e2b0dcc
Sep 15 16:26:52.731203 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:52.731080    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e6809f_8f4b_4eeb_ba75_9ee8226ad2da.slice/crio-5de55e5880d4717c1595696ddfec03ae72e8b63336a8e8d5f4287540678586b8 WatchSource:0}: Error finding container 5de55e5880d4717c1595696ddfec03ae72e8b63336a8e8d5f4287540678586b8: Status 404 returned error can't find the container with id 5de55e5880d4717c1595696ddfec03ae72e8b63336a8e8d5f4287540678586b8
Sep 15 16:26:52.732836 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:52.732476    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac156fa7_e35f_4265_baa3_be8f567a8190.slice/crio-eb25588c6d28883ef5a65db9e920ec2d2e046acf61ab252c6bfefe58a7abd8d9 WatchSource:0}: Error finding container eb25588c6d28883ef5a65db9e920ec2d2e046acf61ab252c6bfefe58a7abd8d9: Status 404 returned error can't find the container with id eb25588c6d28883ef5a65db9e920ec2d2e046acf61ab252c6bfefe58a7abd8d9
Sep 15 16:26:52.732968 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:52.732853    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7764505_fade_42cf_9a24_191b27545755.slice/crio-7681ffe140fe89de91ce5461c5f5e18f6d0d5d5a7d9f1c83f4a11cc981301810 WatchSource:0}: Error finding container 7681ffe140fe89de91ce5461c5f5e18f6d0d5d5a7d9f1c83f4a11cc981301810: Status 404 returned error can't find the container with id 7681ffe140fe89de91ce5461c5f5e18f6d0d5d5a7d9f1c83f4a11cc981301810
Sep 15 16:26:52.735433 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:26:52.735409    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150d8a61_545c_417a_b159_f2e84473250a.slice/crio-35ff128da5e1b3f1176f2c8acd9ee438a8623c506b8d69cba3da1bebb62f08f5 WatchSource:0}: Error finding container 35ff128da5e1b3f1176f2c8acd9ee438a8623c506b8d69cba3da1bebb62f08f5: Status 404 returned error can't find the container with id 35ff128da5e1b3f1176f2c8acd9ee438a8623c506b8d69cba3da1bebb62f08f5
Sep 15 16:26:52.991910 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:52.991881    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs\") pod \"network-metrics-daemon-nvl8x\" (UID: \"9a34ad6e-33f8-4f96-9aca-98998f3bcd62\") " pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:52.992063 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:52.992047    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:52.992141 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:52.992116    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs podName:9a34ad6e-33f8-4f96-9aca-98998f3bcd62 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:54.992095606 +0000 UTC m=+6.036481318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs") pod "network-metrics-daemon-nvl8x" (UID: "9a34ad6e-33f8-4f96-9aca-98998f3bcd62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:53.092754 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.092711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:53.092906 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:53.092885    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:53.092960 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:53.092948    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret podName:22495e12-7c88-4178-9d83-076062a2c424 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:55.092931247 +0000 UTC m=+6.137316962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret") pod "global-pull-secret-syncer-jvr5t" (UID: "22495e12-7c88-4178-9d83-076062a2c424") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:53.193907 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.193872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4nr\" (UniqueName: \"kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr\") pod \"network-check-target-vjsq5\" (UID: \"33198472-1392-416c-a639-0ba97f72fd7f\") " pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:26:53.194088 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:53.194049    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:53.194088 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:53.194073    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:53.194088 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:53.194086    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sb4nr for pod openshift-network-diagnostics/network-check-target-vjsq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:53.194265 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:53.194146    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr podName:33198472-1392-416c-a639-0ba97f72fd7f nodeName:}" failed. No retries permitted until 2026-09-15 16:26:55.194127332 +0000 UTC m=+6.238513061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sb4nr" (UniqueName: "kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr") pod "network-check-target-vjsq5" (UID: "33198472-1392-416c-a639-0ba97f72fd7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:53.427667 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.427560    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 16:21:51 +0000 UTC" deadline="2028-06-23 07:42:22.507863462 +0000 UTC"
Sep 15 16:26:53.427667 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.427628    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15519h15m29.080240573s"
Sep 15 16:26:53.445842 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.444194    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:53.445842 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:53.444344    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:26:53.445842 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.444790    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:53.445842 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:53.444891    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:26:53.445842 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.444961    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:26:53.445842 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:53.445030    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:26:53.465562 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.464834    2565 generic.go:356] "Generic (PLEG): container finished" podID="a675b393087d5d8948209c59db07488b" containerID="07cda9f5a0cb4bc6677c543cfb21033f5a39f613742bd4b76b1634879f65262e" exitCode=0
Sep 15 16:26:53.465562 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.464927    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal" event={"ID":"a675b393087d5d8948209c59db07488b","Type":"ContainerDied","Data":"07cda9f5a0cb4bc6677c543cfb21033f5a39f613742bd4b76b1634879f65262e"}
Sep 15 16:26:53.467696 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.467630    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mj466" event={"ID":"b7764505-fade-42cf-9a24-191b27545755","Type":"ContainerStarted","Data":"7681ffe140fe89de91ce5461c5f5e18f6d0d5d5a7d9f1c83f4a11cc981301810"}
Sep 15 16:26:53.474532 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.474477    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxfh" event={"ID":"150d8a61-545c-417a-b159-f2e84473250a","Type":"ContainerStarted","Data":"35ff128da5e1b3f1176f2c8acd9ee438a8623c506b8d69cba3da1bebb62f08f5"}
Sep 15 16:26:53.483766 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.483740    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fjwgp" event={"ID":"15e6809f-8f4b-4eeb-ba75-9ee8226ad2da","Type":"ContainerStarted","Data":"5de55e5880d4717c1595696ddfec03ae72e8b63336a8e8d5f4287540678586b8"}
Sep 15 16:26:53.490140 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.490114    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kws86" event={"ID":"79a248eb-edf5-486b-b941-217db1abc9ea","Type":"ContainerStarted","Data":"6e0aa40884d372e8e37b8177da5b16affcc05f177a69b588e3c544e5ea633a9a"}
Sep 15 16:26:53.494127 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.494097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7np85" event={"ID":"5a34bc31-c8e6-4214-9762-366d6fe23190","Type":"ContainerStarted","Data":"18bb30af7e32053dbe4e1b7b3e11b1a3f4c709490a4a7319f7d48bc52e7e4c9c"}
Sep 15 16:26:53.496412 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.496386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mnwxq" event={"ID":"32b7532a-7fdd-4341-881f-2715c75f2b88","Type":"ContainerStarted","Data":"bec2332aa20682534942af711ea32bdd205941625c159dc626ba7b8f6f0a39dc"}
Sep 15 16:26:53.501080 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.500830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267" event={"ID":"21dbb64e-2429-48df-bfe2-a7cb863cec30","Type":"ContainerStarted","Data":"6d6717c62afd94b1fbac3c849acac3df77f81bc66c552ebfe67f2d61d4c5b9f1"}
Sep 15 16:26:53.507061 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.507000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-77.ec2.internal" event={"ID":"ce9c1ce3a3fab75c3e888d6b8fff341b","Type":"ContainerStarted","Data":"5e1e56d5861af834c6f0efcbfe1638ada0b66efd2820d36c493cd6e26a231f52"}
Sep 15 16:26:53.513224 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.513197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd" event={"ID":"ac156fa7-e35f-4265-baa3-be8f567a8190","Type":"ContainerStarted","Data":"eb25588c6d28883ef5a65db9e920ec2d2e046acf61ab252c6bfefe58a7abd8d9"}
Sep 15 16:26:53.520894 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:53.520867    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-txbkb" event={"ID":"2f7d621a-2df5-4eb9-b76c-a8e244869670","Type":"ContainerStarted","Data":"8593cbc7eed057cfc2837e8024968004c527097b9923f4b9cfb82f044e2b0dcc"}
Sep 15 16:26:54.539569 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:54.539459    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal" event={"ID":"a675b393087d5d8948209c59db07488b","Type":"ContainerStarted","Data":"c940373c97915cfb1cfe8756c1bd4c2ca7bbcd89a352cdda6d6f9db8a470972c"}
Sep 15 16:26:54.558089 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:54.556775    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-77.ec2.internal" podStartSLOduration=3.556753047 podStartE2EDuration="3.556753047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:26:53.525993846 +0000 UTC m=+4.570379584" watchObservedRunningTime="2026-09-15 16:26:54.556753047 +0000 UTC m=+5.601138783"
Sep 15 16:26:55.007473 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:55.007385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs\") pod \"network-metrics-daemon-nvl8x\" (UID: \"9a34ad6e-33f8-4f96-9aca-98998f3bcd62\") " pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:55.007643 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:55.007574    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:55.007699 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:55.007652    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs podName:9a34ad6e-33f8-4f96-9aca-98998f3bcd62 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:59.007631133 +0000 UTC m=+10.052016851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs") pod "network-metrics-daemon-nvl8x" (UID: "9a34ad6e-33f8-4f96-9aca-98998f3bcd62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:55.107796 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:55.107742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:55.107992 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:55.107970    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:55.108059 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:55.108040    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret podName:22495e12-7c88-4178-9d83-076062a2c424 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:59.108018233 +0000 UTC m=+10.152403950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret") pod "global-pull-secret-syncer-jvr5t" (UID: "22495e12-7c88-4178-9d83-076062a2c424") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:55.208491 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:55.208445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4nr\" (UniqueName: \"kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr\") pod \"network-check-target-vjsq5\" (UID: \"33198472-1392-416c-a639-0ba97f72fd7f\") " pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:26:55.208673 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:55.208625    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:55.208673 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:55.208654    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:55.208769 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:55.208682    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sb4nr for pod openshift-network-diagnostics/network-check-target-vjsq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:55.208846 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:55.208765    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr podName:33198472-1392-416c-a639-0ba97f72fd7f nodeName:}" failed. No retries permitted until 2026-09-15 16:26:59.208729426 +0000 UTC m=+10.253115157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sb4nr" (UniqueName: "kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr") pod "network-check-target-vjsq5" (UID: "33198472-1392-416c-a639-0ba97f72fd7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:55.444319 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:55.444236    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:55.444489 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:55.444388    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:26:55.444558 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:55.444510    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:26:55.444667 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:55.444641    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:26:55.444735 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:55.444721    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:55.444860 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:55.444838    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:26:57.447582 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:57.447547    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:57.448033 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:57.447680    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:26:57.448033 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:57.447769    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:57.448033 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:57.447842    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:26:57.448033 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:57.447887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:26:57.448033 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:57.447943    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:26:59.041962 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:59.041923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs\") pod \"network-metrics-daemon-nvl8x\" (UID: \"9a34ad6e-33f8-4f96-9aca-98998f3bcd62\") " pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:59.042459 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:59.042129    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:59.042459 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:59.042194    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs podName:9a34ad6e-33f8-4f96-9aca-98998f3bcd62 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:07.042175887 +0000 UTC m=+18.086561612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs") pod "network-metrics-daemon-nvl8x" (UID: "9a34ad6e-33f8-4f96-9aca-98998f3bcd62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:59.142819 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:59.142669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:59.143020 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:59.142831    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:59.143020 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:59.142911    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret podName:22495e12-7c88-4178-9d83-076062a2c424 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:07.142889071 +0000 UTC m=+18.187274789 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret") pod "global-pull-secret-syncer-jvr5t" (UID: "22495e12-7c88-4178-9d83-076062a2c424") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:59.243212 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:59.243165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4nr\" (UniqueName: \"kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr\") pod \"network-check-target-vjsq5\" (UID: \"33198472-1392-416c-a639-0ba97f72fd7f\") " pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:26:59.243390 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:59.243321    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:59.243390 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:59.243348    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:59.243390 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:59.243362    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sb4nr for pod openshift-network-diagnostics/network-check-target-vjsq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:59.243552 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:59.243425    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr podName:33198472-1392-416c-a639-0ba97f72fd7f nodeName:}" failed. No retries permitted until 2026-09-15 16:27:07.243406494 +0000 UTC m=+18.287792219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sb4nr" (UniqueName: "kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr") pod "network-check-target-vjsq5" (UID: "33198472-1392-416c-a639-0ba97f72fd7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:59.448368 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:59.447589    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:26:59.448368 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:59.447720    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:26:59.448368 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:59.448100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:26:59.448368 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:59.448201    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:26:59.448368 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:26:59.448252    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:26:59.448368 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:26:59.448322    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:01.444759 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:01.444275    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:01.444759 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:01.444275    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:01.444759 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:01.444536    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:27:01.444759 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:01.444417    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:01.444759 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:01.444609    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:01.444759 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:01.444702    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:27:03.444005 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:03.443980    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:03.444005 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:03.443992    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:03.444501 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:03.444113    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:27:03.444501 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:03.444199    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:03.444501 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:03.444196    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:03.444501 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:03.444294    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:27:05.444475 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:05.444433    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:05.444957 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:05.444566    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:05.444957 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:05.444584    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:05.444957 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:05.444588    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:27:05.444957 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:05.444681    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:05.444957 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:05.444791    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:27:07.103941 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:07.103908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs\") pod \"network-metrics-daemon-nvl8x\" (UID: \"9a34ad6e-33f8-4f96-9aca-98998f3bcd62\") " pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:07.104384 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:07.104059    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:07.104384 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:07.104131    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs podName:9a34ad6e-33f8-4f96-9aca-98998f3bcd62 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:23.104109201 +0000 UTC m=+34.148494918 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs") pod "network-metrics-daemon-nvl8x" (UID: "9a34ad6e-33f8-4f96-9aca-98998f3bcd62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:07.204234 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:07.204199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:07.204399 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:07.204343    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:07.204471 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:07.204403    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret podName:22495e12-7c88-4178-9d83-076062a2c424 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:23.204387969 +0000 UTC m=+34.248773682 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret") pod "global-pull-secret-syncer-jvr5t" (UID: "22495e12-7c88-4178-9d83-076062a2c424") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:07.304940 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:07.304903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4nr\" (UniqueName: \"kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr\") pod \"network-check-target-vjsq5\" (UID: \"33198472-1392-416c-a639-0ba97f72fd7f\") " pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:07.305132 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:07.305105    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:27:07.305132 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:07.305131    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:27:07.305331 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:07.305145    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sb4nr for pod openshift-network-diagnostics/network-check-target-vjsq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:07.305331 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:07.305211    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr podName:33198472-1392-416c-a639-0ba97f72fd7f nodeName:}" failed. No retries permitted until 2026-09-15 16:27:23.305191304 +0000 UTC m=+34.349577033 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sb4nr" (UniqueName: "kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr") pod "network-check-target-vjsq5" (UID: "33198472-1392-416c-a639-0ba97f72fd7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:07.444034 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:07.443947    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:07.444034 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:07.443993    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:07.444244 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:07.443953    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:07.444244 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:07.444082    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:27:07.444244 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:07.444158    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:07.444244 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:07.444224    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:27:09.444884 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:09.444852    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:09.445317 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:09.444954    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:09.445317 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:09.444965    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:09.445317 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:09.444996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:09.445317 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:09.445089    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:27:09.445317 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:09.445175    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:27:11.444577 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.444346    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:11.445617 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.444344    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:11.445617 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:11.444678    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:11.445617 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.444419    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:11.445617 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:11.444736    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:27:11.445617 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:11.444859    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:27:11.585007 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.584970    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mj466" event={"ID":"b7764505-fade-42cf-9a24-191b27545755","Type":"ContainerStarted","Data":"ae67079e297f0d2b5991953ce257922dc7f824c1facfd1fd6635ff792bba1fca"}
Sep 15 16:27:11.586658 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.586632    2565 generic.go:356] "Generic (PLEG): container finished" podID="150d8a61-545c-417a-b159-f2e84473250a" containerID="ba88a828647349f4636b0daf69dbc6bfbc27bf191256cb1929f864e9e48b22e2" exitCode=0
Sep 15 16:27:11.586799 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.586722    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxfh" event={"ID":"150d8a61-545c-417a-b159-f2e84473250a","Type":"ContainerDied","Data":"ba88a828647349f4636b0daf69dbc6bfbc27bf191256cb1929f864e9e48b22e2"}
Sep 15 16:27:11.588350 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.588331    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fjwgp" event={"ID":"15e6809f-8f4b-4eeb-ba75-9ee8226ad2da","Type":"ContainerStarted","Data":"f5cbe09b368bfb95e072edf24125a77648f0cc4fc9c1793e314e172bc5358f2f"}
Sep 15 16:27:11.590656 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.590628    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7np85" event={"ID":"5a34bc31-c8e6-4214-9762-366d6fe23190","Type":"ContainerStarted","Data":"4df444aca402938897681a2688d3ac66f4eab068a196b557ceb2e384f00950fc"}
Sep 15 16:27:11.592358 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.592338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mnwxq" event={"ID":"32b7532a-7fdd-4341-881f-2715c75f2b88","Type":"ContainerStarted","Data":"9e26e4536a06c600e3047971e5bdeee4e57e1e5add8630b1b4dffea347c1d9a2"}
Sep 15 16:27:11.593826 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.593790    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267" event={"ID":"21dbb64e-2429-48df-bfe2-a7cb863cec30","Type":"ContainerStarted","Data":"b4b531042b4ef51d8a3d297192861bf8ccaaff4a41b9e1ad8742aafe7c8a6230"}
Sep 15 16:27:11.596904 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.596876    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd" event={"ID":"ac156fa7-e35f-4265-baa3-be8f567a8190","Type":"ContainerStarted","Data":"68d70f04c14b706f52b520b0740a585fde1c63a6d26c46c052aff27e8d2a7c91"}
Sep 15 16:27:11.597000 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.596948    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd" event={"ID":"ac156fa7-e35f-4265-baa3-be8f567a8190","Type":"ContainerStarted","Data":"f81bfe44b3dba6237e0074a1369a6bbe6202b9daab5b7955f97a8c2cdee28dae"}
Sep 15 16:27:11.597065 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.597000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd" event={"ID":"ac156fa7-e35f-4265-baa3-be8f567a8190","Type":"ContainerStarted","Data":"0c9095f0cf8b33e64ec1f6a3f826c2f3642ca54b67c1975150c90642f717b163"}
Sep 15 16:27:11.597065 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.597049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd" event={"ID":"ac156fa7-e35f-4265-baa3-be8f567a8190","Type":"ContainerStarted","Data":"7c62ba1ee461a4e1a268b1d3272a33a8f87413a5a8b1678e525886a65be69a5a"}
Sep 15 16:27:11.597167 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.597096    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd" event={"ID":"ac156fa7-e35f-4265-baa3-be8f567a8190","Type":"ContainerStarted","Data":"1b236e576361bdba6737a43db3315310fafaec49a79460cc54af7e50e0b80242"}
Sep 15 16:27:11.597167 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.597145    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd" event={"ID":"ac156fa7-e35f-4265-baa3-be8f567a8190","Type":"ContainerStarted","Data":"9afcaa57d9cea12bef2117287e64d5acf7430b8ece93712ac1b9dbacd2cc74cf"}
Sep 15 16:27:11.598285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.598267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-txbkb" event={"ID":"2f7d621a-2df5-4eb9-b76c-a8e244869670","Type":"ContainerStarted","Data":"91df439fffa28f7e802b98b89e0ae1a956ceae5348d169397eae4e732a9f0c75"}
Sep 15 16:27:11.613657 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.613612    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mj466" podStartSLOduration=3.84730656 podStartE2EDuration="21.613597746s" totalImagesPullingTime="17.766291186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:11.612810061 +0000 UTC m=+22.657195794" watchObservedRunningTime="2026-09-15 16:27:11.613597746 +0000 UTC m=+22.657983479"
Sep 15 16:27:11.613796 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.613772    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-77.ec2.internal" podStartSLOduration=20.613765202 podStartE2EDuration="20.613765202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:26:54.558051294 +0000 UTC m=+5.602437044" watchObservedRunningTime="2026-09-15 16:27:11.613765202 +0000 UTC m=+22.658150935"
Sep 15 16:27:11.653237 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.653200    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7np85" podStartSLOduration=4.881772315 podStartE2EDuration="22.653186326s" totalImagesPullingTime="17.771414011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:11.630285951 +0000 UTC m=+22.674671682" watchObservedRunningTime="2026-09-15 16:27:11.653186326 +0000 UTC m=+22.697572088"
Sep 15 16:27:11.678759 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.678723    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fjwgp" podStartSLOduration=12.071228418 podStartE2EDuration="22.678695053s" totalImagesPullingTime="10.607466635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:11.678361407 +0000 UTC m=+22.722747139" watchObservedRunningTime="2026-09-15 16:27:11.678695053 +0000 UTC m=+22.723080775"
Sep 15 16:27:11.678856 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.678823    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mnwxq" podStartSLOduration=3.899727355 podStartE2EDuration="21.678819352s" totalImagesPullingTime="17.779091997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:11.653087204 +0000 UTC m=+22.697472939" watchObservedRunningTime="2026-09-15 16:27:11.678819352 +0000 UTC m=+22.723205083"
Sep 15 16:27:11.731437 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.731353    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-txbkb" podStartSLOduration=4.899723295 podStartE2EDuration="22.731338475s" totalImagesPullingTime="17.83161518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:11.703865852 +0000 UTC m=+22.748251583" watchObservedRunningTime="2026-09-15 16:27:11.731338475 +0000 UTC m=+22.775724206"
Sep 15 16:27:11.935578 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:11.935554    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 16:27:12.414456 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:12.414324    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T16:27:11.935573456Z","UUID":"ac1b7c1f-07e2-4df4-aa49-834de75076f2","Handler":null,"Name":"","Endpoint":""}
Sep 15 16:27:12.416123 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:12.416077    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 16:27:12.416123 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:12.416111    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 16:27:12.603885 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:12.603837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kws86" event={"ID":"79a248eb-edf5-486b-b941-217db1abc9ea","Type":"ContainerStarted","Data":"ececc0c8e7954bf6daf40746bac5df405ca3d755bcc136bc98cc92e7288723c2"}
Sep 15 16:27:12.606624 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:12.606593    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267" event={"ID":"21dbb64e-2429-48df-bfe2-a7cb863cec30","Type":"ContainerStarted","Data":"a875369b717fbc4f230194a90bcfa2de22ad685a239da0a7907f9c7e47ce26cf"}
Sep 15 16:27:12.621342 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:12.621294    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kws86" podStartSLOduration=4.849358129 podStartE2EDuration="22.621275522s" totalImagesPullingTime="17.771917393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:12.620881058 +0000 UTC m=+23.665266820" watchObservedRunningTime="2026-09-15 16:27:12.621275522 +0000 UTC m=+23.665661256"
Sep 15 16:27:13.443656 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:13.443617    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:13.443836 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:13.443667    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:13.443836 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:13.443616    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:13.443836 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:13.443761    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:27:13.443999 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:13.443848    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:13.443999 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:13.443932    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:27:13.611956 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:13.611873    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267" event={"ID":"21dbb64e-2429-48df-bfe2-a7cb863cec30","Type":"ContainerStarted","Data":"da33423e80767aea6570927853281d45b9d7e58daa07b0e669e931fb27338d8c"}
Sep 15 16:27:13.615969 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:13.615926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd" event={"ID":"ac156fa7-e35f-4265-baa3-be8f567a8190","Type":"ContainerStarted","Data":"e800571a6fc33c865b8544da46d3c8d119dcd90f44df821fcba0c6c26b946864"}
Sep 15 16:27:13.631459 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:13.631412    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p267" podStartSLOduration=4.867519431 podStartE2EDuration="24.631396525s" totalImagesPullingTime="19.763877094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:13.630997957 +0000 UTC m=+24.675383700" watchObservedRunningTime="2026-09-15 16:27:13.631396525 +0000 UTC m=+24.675782256"
Sep 15 16:27:15.444146 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:15.443912    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:15.444573 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:15.443914    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:15.444573 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:15.444308    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:27:15.444573 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:15.443970    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:15.444573 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:15.444385    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:15.444573 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:15.444510    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:27:15.625471 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:15.625436    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd" event={"ID":"ac156fa7-e35f-4265-baa3-be8f567a8190","Type":"ContainerStarted","Data":"8ddcd198788860ca0fe5f543e84d76a9d292d07a9ae678c394222127bd4152fc"}
Sep 15 16:27:15.625819 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:15.625722    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:27:15.625819 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:15.625810    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:27:15.645491 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:15.645469    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:27:15.691293 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:15.691233    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd" podStartSLOduration=8.869919739 podStartE2EDuration="26.691209335s" totalImagesPullingTime="17.821289596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:15.664133703 +0000 UTC m=+26.708519435" watchObservedRunningTime="2026-09-15 16:27:15.691209335 +0000 UTC m=+26.735595068"
Sep 15 16:27:16.416431 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:16.416250    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fjwgp"
Sep 15 16:27:16.416908 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:16.416889    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fjwgp"
Sep 15 16:27:16.629437 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:16.629407    2565 generic.go:356] "Generic (PLEG): container finished" podID="150d8a61-545c-417a-b159-f2e84473250a" containerID="1acbf47313fae9c1ca8b8af68810c516027558826daabcc37541315c0828a0aa" exitCode=0
Sep 15 16:27:16.630394 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:16.629491    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxfh" event={"ID":"150d8a61-545c-417a-b159-f2e84473250a","Type":"ContainerDied","Data":"1acbf47313fae9c1ca8b8af68810c516027558826daabcc37541315c0828a0aa"}
Sep 15 16:27:16.630394 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:16.629625    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fjwgp"
Sep 15 16:27:16.630394 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:16.630314    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:27:16.630983 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:16.630408    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fjwgp"
Sep 15 16:27:16.645329 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:16.645306    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:27:17.444534 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:17.444502    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:17.444808 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:17.444517    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:17.444808 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:17.444606    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:27:17.444808 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:17.444731    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:27:17.444808 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:17.444766    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:17.444999 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:17.444848    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:17.552001 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:17.551903    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vjsq5"]
Sep 15 16:27:17.552864 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:17.552795    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jvr5t"]
Sep 15 16:27:17.553524 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:17.553502    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nvl8x"]
Sep 15 16:27:17.632267 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:17.632241    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:17.632267 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:17.632271    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:17.632755 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:17.632285    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:17.632755 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:17.632492    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:27:17.632755 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:17.632600    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:17.632755 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:17.632679    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:27:17.880142 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:17.880065    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:17.975432 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:17.975400    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:18.636813 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:18.636735    2565 generic.go:356] "Generic (PLEG): container finished" podID="150d8a61-545c-417a-b159-f2e84473250a" containerID="c95d2f7c8c2f4d9912fec752b0c881a56f9c620c857345696724328d40494aa6" exitCode=0
Sep 15 16:27:18.637160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:18.636812    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxfh" event={"ID":"150d8a61-545c-417a-b159-f2e84473250a","Type":"ContainerDied","Data":"c95d2f7c8c2f4d9912fec752b0c881a56f9c620c857345696724328d40494aa6"}
Sep 15 16:27:19.444957 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:19.444930    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:19.445099 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:19.445015    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:19.445099 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:19.445074    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:27:19.445272 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:19.445116    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:19.445343 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:19.445247    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:27:19.445409 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:19.445368    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:19.641558 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:19.641525    2565 generic.go:356] "Generic (PLEG): container finished" podID="150d8a61-545c-417a-b159-f2e84473250a" containerID="03c8e82a23604dd540782ff0530f25e27c40b31c4a7777f415015ddfc0712233" exitCode=0
Sep 15 16:27:19.641939 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:19.641595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxfh" event={"ID":"150d8a61-545c-417a-b159-f2e84473250a","Type":"ContainerDied","Data":"03c8e82a23604dd540782ff0530f25e27c40b31c4a7777f415015ddfc0712233"}
Sep 15 16:27:21.444562 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:21.444525    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:21.445250 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:21.444535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:21.445250 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:21.444661    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:21.445250 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:21.444535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:21.445250 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:21.444779    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:27:21.445250 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:21.444861    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:27:22.853959 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:22.853918    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:22.913684 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:22.913628    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:23.112770 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:23.112678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs\") pod \"network-metrics-daemon-nvl8x\" (UID: \"9a34ad6e-33f8-4f96-9aca-98998f3bcd62\") " pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:23.112939 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:23.112850    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:23.112939 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:23.112930    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs podName:9a34ad6e-33f8-4f96-9aca-98998f3bcd62 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:55.112909635 +0000 UTC m=+66.157295348 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs") pod "network-metrics-daemon-nvl8x" (UID: "9a34ad6e-33f8-4f96-9aca-98998f3bcd62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:23.213324 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:23.213281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:23.213491 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:23.213436    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:23.213539 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:23.213502    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret podName:22495e12-7c88-4178-9d83-076062a2c424 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:55.213483331 +0000 UTC m=+66.257869055 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret") pod "global-pull-secret-syncer-jvr5t" (UID: "22495e12-7c88-4178-9d83-076062a2c424") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:23.314277 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:23.314236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4nr\" (UniqueName: \"kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr\") pod \"network-check-target-vjsq5\" (UID: \"33198472-1392-416c-a639-0ba97f72fd7f\") " pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:23.314453 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:23.314436    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:27:23.314509 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:23.314461    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:27:23.314509 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:23.314473    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sb4nr for pod openshift-network-diagnostics/network-check-target-vjsq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:23.314604 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:23.314535    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr podName:33198472-1392-416c-a639-0ba97f72fd7f nodeName:}" failed. No retries permitted until 2026-09-15 16:27:55.314518378 +0000 UTC m=+66.358904104 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sb4nr" (UniqueName: "kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr") pod "network-check-target-vjsq5" (UID: "33198472-1392-416c-a639-0ba97f72fd7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:23.444197 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:23.444163    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:23.444197 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:23.444192    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:23.444472 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:23.444177    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:23.444472 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:23.444299    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jvr5t" podUID="22495e12-7c88-4178-9d83-076062a2c424"
Sep 15 16:27:23.444472 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:23.444369    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vjsq5" podUID="33198472-1392-416c-a639-0ba97f72fd7f"
Sep 15 16:27:23.444472 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:23.444458    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nvl8x" podUID="9a34ad6e-33f8-4f96-9aca-98998f3bcd62"
Sep 15 16:27:24.287639 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.287617    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-77.ec2.internal" event="NodeReady"
Sep 15 16:27:24.288300 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.287764    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 16:27:24.337219 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.337182    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kdqvv"]
Sep 15 16:27:24.355176 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.355152    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-czx4f"]
Sep 15 16:27:24.355324 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.355301    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kdqvv"
Sep 15 16:27:24.358355 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.358335    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-55j2f\""
Sep 15 16:27:24.358355 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.358348    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 16:27:24.358509 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.358369    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 16:27:24.358509 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.358467    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 16:27:24.369178 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.369160    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kdqvv"]
Sep 15 16:27:24.369260 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.369188    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-czx4f"]
Sep 15 16:27:24.369301 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.369264    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:24.371956 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.371935    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xg4rz\""
Sep 15 16:27:24.372203 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.372186    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 16:27:24.372282 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.372225    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 16:27:24.523375 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.523283    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59csf\" (UniqueName: \"kubernetes.io/projected/aaa40fc5-09b9-4768-95b9-005e46e994ed-kube-api-access-59csf\") pod \"ingress-canary-kdqvv\" (UID: \"aaa40fc5-09b9-4768-95b9-005e46e994ed\") " pod="openshift-ingress-canary/ingress-canary-kdqvv"
Sep 15 16:27:24.523375 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.523323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6jn8\" (UniqueName: \"kubernetes.io/projected/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-kube-api-access-w6jn8\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:24.523566 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.523381    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaa40fc5-09b9-4768-95b9-005e46e994ed-cert\") pod \"ingress-canary-kdqvv\" (UID: \"aaa40fc5-09b9-4768-95b9-005e46e994ed\") " pod="openshift-ingress-canary/ingress-canary-kdqvv"
Sep 15 16:27:24.523566 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.523411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-tmp-dir\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:24.523566 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.523456    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-config-volume\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:24.523566 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.523486    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:24.624553 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.624516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59csf\" (UniqueName: \"kubernetes.io/projected/aaa40fc5-09b9-4768-95b9-005e46e994ed-kube-api-access-59csf\") pod \"ingress-canary-kdqvv\" (UID: \"aaa40fc5-09b9-4768-95b9-005e46e994ed\") " pod="openshift-ingress-canary/ingress-canary-kdqvv"
Sep 15 16:27:24.624553 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.624549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6jn8\" (UniqueName: \"kubernetes.io/projected/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-kube-api-access-w6jn8\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:24.624826 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.624572    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaa40fc5-09b9-4768-95b9-005e46e994ed-cert\") pod \"ingress-canary-kdqvv\" (UID: \"aaa40fc5-09b9-4768-95b9-005e46e994ed\") " pod="openshift-ingress-canary/ingress-canary-kdqvv"
Sep 15 16:27:24.624826 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.624587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-tmp-dir\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:24.624826 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.624616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-config-volume\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:24.624826 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.624760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:24.625599 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:24.625571    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 16:27:24.625599 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.625589    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-tmp-dir\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:24.625798 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:24.625661    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls podName:ef4b4ee7-812d-4ed4-9225-c2f5616c8133 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:25.125638802 +0000 UTC m=+36.170024513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls") pod "dns-default-czx4f" (UID: "ef4b4ee7-812d-4ed4-9225-c2f5616c8133") : secret "dns-default-metrics-tls" not found
Sep 15 16:27:24.626193 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.625884    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-config-volume\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:24.631941 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.631914    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaa40fc5-09b9-4768-95b9-005e46e994ed-cert\") pod \"ingress-canary-kdqvv\" (UID: \"aaa40fc5-09b9-4768-95b9-005e46e994ed\") " pod="openshift-ingress-canary/ingress-canary-kdqvv"
Sep 15 16:27:24.632841 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.632818    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59csf\" (UniqueName: \"kubernetes.io/projected/aaa40fc5-09b9-4768-95b9-005e46e994ed-kube-api-access-59csf\") pod \"ingress-canary-kdqvv\" (UID: \"aaa40fc5-09b9-4768-95b9-005e46e994ed\") " pod="openshift-ingress-canary/ingress-canary-kdqvv"
Sep 15 16:27:24.633756 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.633736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6jn8\" (UniqueName: \"kubernetes.io/projected/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-kube-api-access-w6jn8\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:24.656203 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.656171    2565 generic.go:356] "Generic (PLEG): container finished" podID="150d8a61-545c-417a-b159-f2e84473250a" containerID="21f6a5fa85dd026524eb866190c19e3e9c1ad3ad46b813ac3c474e0123a6fd19" exitCode=0
Sep 15 16:27:24.656294 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.656241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxfh" event={"ID":"150d8a61-545c-417a-b159-f2e84473250a","Type":"ContainerDied","Data":"21f6a5fa85dd026524eb866190c19e3e9c1ad3ad46b813ac3c474e0123a6fd19"}
Sep 15 16:27:24.664430 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.664413    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kdqvv"
Sep 15 16:27:24.993170 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:24.992958    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kdqvv"]
Sep 15 16:27:24.998347 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:27:24.998325    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa40fc5_09b9_4768_95b9_005e46e994ed.slice/crio-b55b8cb36f304adc1f379c095f78fd6bf1edef8e637ded726a09bc866c9112ef WatchSource:0}: Error finding container b55b8cb36f304adc1f379c095f78fd6bf1edef8e637ded726a09bc866c9112ef: Status 404 returned error can't find the container with id b55b8cb36f304adc1f379c095f78fd6bf1edef8e637ded726a09bc866c9112ef
Sep 15 16:27:25.032126 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.032096    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:25.088950 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.088880    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:25.129956 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.129926    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:25.130079 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:25.130027    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 16:27:25.130079 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:25.130076    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls podName:ef4b4ee7-812d-4ed4-9225-c2f5616c8133 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:26.130061939 +0000 UTC m=+37.174447652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls") pod "dns-default-czx4f" (UID: "ef4b4ee7-812d-4ed4-9225-c2f5616c8133") : secret "dns-default-metrics-tls" not found
Sep 15 16:27:25.443743 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.443696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:25.443743 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.443726    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:25.444319 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.443726    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:25.446888 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.446864    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 16:27:25.447796 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.447773    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 16:27:25.448075 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.448053    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 16:27:25.448075 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.448054    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vcckz\""
Sep 15 16:27:25.448221 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.448054    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9w9qq\""
Sep 15 16:27:25.448221 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.448088    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 16:27:25.660172 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.660137    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kdqvv" event={"ID":"aaa40fc5-09b9-4768-95b9-005e46e994ed","Type":"ContainerStarted","Data":"b55b8cb36f304adc1f379c095f78fd6bf1edef8e637ded726a09bc866c9112ef"}
Sep 15 16:27:25.663933 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.663901    2565 generic.go:356] "Generic (PLEG): container finished" podID="150d8a61-545c-417a-b159-f2e84473250a" containerID="2fec8ba0fddd83856e6ee0b5f398407983ef47a3744ebeaf8a66192e5789ae9e" exitCode=0
Sep 15 16:27:25.664121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:25.663989    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxfh" event={"ID":"150d8a61-545c-417a-b159-f2e84473250a","Type":"ContainerDied","Data":"2fec8ba0fddd83856e6ee0b5f398407983ef47a3744ebeaf8a66192e5789ae9e"}
Sep 15 16:27:26.138626 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:26.138545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:26.138791 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:26.138720    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 16:27:26.138858 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:26.138796    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls podName:ef4b4ee7-812d-4ed4-9225-c2f5616c8133 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:28.138777443 +0000 UTC m=+39.183163282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls") pod "dns-default-czx4f" (UID: "ef4b4ee7-812d-4ed4-9225-c2f5616c8133") : secret "dns-default-metrics-tls" not found
Sep 15 16:27:26.671253 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:26.671054    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxxfh" event={"ID":"150d8a61-545c-417a-b159-f2e84473250a","Type":"ContainerStarted","Data":"ee030c81a45bf779f4bd6ee979c46abf3247cf08e5ff9700794ee4040791fbb8"}
Sep 15 16:27:26.703351 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:26.703291    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cxxfh" podStartSLOduration=9.922891439 podStartE2EDuration="37.703274776s" totalImagesPullingTime="27.780383337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:26.702672431 +0000 UTC m=+37.747058229" watchObservedRunningTime="2026-09-15 16:27:26.703274776 +0000 UTC m=+37.747660507"
Sep 15 16:27:26.735492 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:26.735459    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:26.836226 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:26.836188    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:26.936919 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:26.936796    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:27.005414 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:27.005378    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:27.387993 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:27.387924    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:27.675613 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:27.675578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kdqvv" event={"ID":"aaa40fc5-09b9-4768-95b9-005e46e994ed","Type":"ContainerStarted","Data":"d51fbab96576ccca44efb6268fc810f08a0f426eb98dbc9f79a34f83f86f8d52"}
Sep 15 16:27:27.694229 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:27.694182    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kdqvv" podStartSLOduration=1.221827983 podStartE2EDuration="3.694166232s" totalImagesPullingTime="2.472338249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:27.693654783 +0000 UTC m=+38.738040516" watchObservedRunningTime="2026-09-15 16:27:27.694166232 +0000 UTC m=+38.738551987"
Sep 15 16:27:28.154296 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:28.154262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:28.154469 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:28.154407    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 16:27:28.154514 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:28.154472    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls podName:ef4b4ee7-812d-4ed4-9225-c2f5616c8133 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:32.154459529 +0000 UTC m=+43.198845238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls") pod "dns-default-czx4f" (UID: "ef4b4ee7-812d-4ed4-9225-c2f5616c8133") : secret "dns-default-metrics-tls" not found
Sep 15 16:27:28.861519 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:28.861484    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:28.966561 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:28.966531    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:29.335281 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.335249    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:29.437068 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.437032    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:29.546453 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.546425    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:29.653362 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.653333    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:29.778161 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.778128    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mwbnk"]
Sep 15 16:27:29.801248 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.801215    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:29.801836 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.801811    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mwbnk"]
Sep 15 16:27:29.801959 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.801918    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:29.805622 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.805588    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 16:27:29.805750 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.805635    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jpbwt\""
Sep 15 16:27:29.805750 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.805649    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 16:27:29.806171 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.806153    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 16:27:29.806171 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.806167    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 16:27:29.967036 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.966956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5z4\" (UniqueName: \"kubernetes.io/projected/148e3028-957d-460a-abfe-e936406d0989-kube-api-access-4j5z4\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:29.967036 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.967015    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/148e3028-957d-460a-abfe-e936406d0989-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:29.967390 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.967110    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/148e3028-957d-460a-abfe-e936406d0989-data-volume\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:29.967390 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.967142    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:29.967390 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:29.967163    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/148e3028-957d-460a-abfe-e936406d0989-crio-socket\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:30.068358 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.068319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/148e3028-957d-460a-abfe-e936406d0989-data-volume\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:30.068358 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.068362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:30.068505 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.068385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/148e3028-957d-460a-abfe-e936406d0989-crio-socket\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:30.068505 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.068405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5z4\" (UniqueName: \"kubernetes.io/projected/148e3028-957d-460a-abfe-e936406d0989-kube-api-access-4j5z4\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:30.068505 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.068446    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/148e3028-957d-460a-abfe-e936406d0989-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:30.068598 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:30.068528    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:30.068598 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:30.068567    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 16:27:30.068656 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.068592    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/148e3028-957d-460a-abfe-e936406d0989-crio-socket\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:30.068656 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:30.068613    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls podName:148e3028-957d-460a-abfe-e936406d0989 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:30.568587545 +0000 UTC m=+41.612973269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwbnk" (UID: "148e3028-957d-460a-abfe-e936406d0989") : secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:30.068656 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:30.068634    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/148e3028-957d-460a-abfe-e936406d0989-kube-rbac-proxy-cm podName:148e3028-957d-460a-abfe-e936406d0989 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:30.568621982 +0000 UTC m=+41.613007696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/148e3028-957d-460a-abfe-e936406d0989-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwbnk" (UID: "148e3028-957d-460a-abfe-e936406d0989") : configmap "kube-rbac-proxy" not found
Sep 15 16:27:30.068789 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.068658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/148e3028-957d-460a-abfe-e936406d0989-data-volume\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:30.081760 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.081734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5z4\" (UniqueName: \"kubernetes.io/projected/148e3028-957d-460a-abfe-e936406d0989-kube-api-access-4j5z4\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:30.571982 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.571944    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:30.572160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.572007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/148e3028-957d-460a-abfe-e936406d0989-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:30.572160 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:30.572083    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 16:27:30.572160 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:30.572096    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:30.572160 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:30.572134    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/148e3028-957d-460a-abfe-e936406d0989-kube-rbac-proxy-cm podName:148e3028-957d-460a-abfe-e936406d0989 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:31.572120088 +0000 UTC m=+42.616505801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/148e3028-957d-460a-abfe-e936406d0989-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwbnk" (UID: "148e3028-957d-460a-abfe-e936406d0989") : configmap "kube-rbac-proxy" not found
Sep 15 16:27:30.572160 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:30.572155    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls podName:148e3028-957d-460a-abfe-e936406d0989 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:31.572141979 +0000 UTC m=+42.616527702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwbnk" (UID: "148e3028-957d-460a-abfe-e936406d0989") : secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:30.585123 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.585097    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-d4pf4"]
Sep 15 16:27:30.607907 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.607880    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-d4pf4"
Sep 15 16:27:30.609908 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.609874    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-d4pf4"]
Sep 15 16:27:30.611297 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.611275    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4x6z7\""
Sep 15 16:27:30.611418 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.611297    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 15 16:27:30.612007 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.611993    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 15 16:27:30.773818 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.773784    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qpxd\" (UniqueName: \"kubernetes.io/projected/2bd47ace-dade-4b8d-af86-a27937a4ac38-kube-api-access-5qpxd\") pod \"migrator-8b4d697c5-d4pf4\" (UID: \"2bd47ace-dade-4b8d-af86-a27937a4ac38\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-d4pf4"
Sep 15 16:27:30.874345 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.874274    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qpxd\" (UniqueName: \"kubernetes.io/projected/2bd47ace-dade-4b8d-af86-a27937a4ac38-kube-api-access-5qpxd\") pod \"migrator-8b4d697c5-d4pf4\" (UID: \"2bd47ace-dade-4b8d-af86-a27937a4ac38\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-d4pf4"
Sep 15 16:27:30.883544 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.883514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qpxd\" (UniqueName: \"kubernetes.io/projected/2bd47ace-dade-4b8d-af86-a27937a4ac38-kube-api-access-5qpxd\") pod \"migrator-8b4d697c5-d4pf4\" (UID: \"2bd47ace-dade-4b8d-af86-a27937a4ac38\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-d4pf4"
Sep 15 16:27:30.917446 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:30.917419    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-d4pf4"
Sep 15 16:27:31.191448 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.191419    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:31.203926 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.203901    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-d4pf4"]
Sep 15 16:27:31.207569 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:27:31.207544    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd47ace_dade_4b8d_af86_a27937a4ac38.slice/crio-45ce7c7ec56914c355c73211dcd55cad42d12e76e0023c073da0782d4f1aadb9 WatchSource:0}: Error finding container 45ce7c7ec56914c355c73211dcd55cad42d12e76e0023c073da0782d4f1aadb9: Status 404 returned error can't find the container with id 45ce7c7ec56914c355c73211dcd55cad42d12e76e0023c073da0782d4f1aadb9
Sep 15 16:27:31.300358 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.300330    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:31.397246 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.397221    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:31.579605 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.579514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:31.579605 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.579579    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/148e3028-957d-460a-abfe-e936406d0989-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:31.579817 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:31.579661    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 16:27:31.579817 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:31.579662    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:31.579817 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:31.579740    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/148e3028-957d-460a-abfe-e936406d0989-kube-rbac-proxy-cm podName:148e3028-957d-460a-abfe-e936406d0989 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:33.579725336 +0000 UTC m=+44.624111058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/148e3028-957d-460a-abfe-e936406d0989-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwbnk" (UID: "148e3028-957d-460a-abfe-e936406d0989") : configmap "kube-rbac-proxy" not found
Sep 15 16:27:31.579817 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:31.579769    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls podName:148e3028-957d-460a-abfe-e936406d0989 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:33.579755829 +0000 UTC m=+44.624141544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwbnk" (UID: "148e3028-957d-460a-abfe-e936406d0989") : secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:31.685778 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.685744    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-d4pf4" event={"ID":"2bd47ace-dade-4b8d-af86-a27937a4ac38","Type":"ContainerStarted","Data":"45ce7c7ec56914c355c73211dcd55cad42d12e76e0023c073da0782d4f1aadb9"}
Sep 15 16:27:31.721649 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.721628    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mj466_b7764505-fade-42cf-9a24-191b27545755/dns-node-resolver/0.log"
Sep 15 16:27:31.852839 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.852761    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-g7849"]
Sep 15 16:27:31.869242 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.869204    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-g7849"]
Sep 15 16:27:31.869429 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.869321    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-g7849"
Sep 15 16:27:31.872653 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.872631    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 15 16:27:31.873252 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.873232    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 15 16:27:31.873621 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.873600    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 15 16:27:31.873736 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.873640    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 15 16:27:31.873736 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.873607    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6pbxt\""
Sep 15 16:27:31.889872 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.889851    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:31.976228 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.976178    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:31.982241 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.982218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f80c0763-ded0-4716-b424-6d10dcfa1c24-signing-key\") pod \"service-ca-568869fdc5-g7849\" (UID: \"f80c0763-ded0-4716-b424-6d10dcfa1c24\") " pod="openshift-service-ca/service-ca-568869fdc5-g7849"
Sep 15 16:27:31.982397 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.982365    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wmk2\" (UniqueName: \"kubernetes.io/projected/f80c0763-ded0-4716-b424-6d10dcfa1c24-kube-api-access-4wmk2\") pod \"service-ca-568869fdc5-g7849\" (UID: \"f80c0763-ded0-4716-b424-6d10dcfa1c24\") " pod="openshift-service-ca/service-ca-568869fdc5-g7849"
Sep 15 16:27:31.982456 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:31.982419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f80c0763-ded0-4716-b424-6d10dcfa1c24-signing-cabundle\") pod \"service-ca-568869fdc5-g7849\" (UID: \"f80c0763-ded0-4716-b424-6d10dcfa1c24\") " pod="openshift-service-ca/service-ca-568869fdc5-g7849"
Sep 15 16:27:32.082967 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:32.082932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f80c0763-ded0-4716-b424-6d10dcfa1c24-signing-key\") pod \"service-ca-568869fdc5-g7849\" (UID: \"f80c0763-ded0-4716-b424-6d10dcfa1c24\") " pod="openshift-service-ca/service-ca-568869fdc5-g7849"
Sep 15 16:27:32.083107 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:32.083044    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wmk2\" (UniqueName: \"kubernetes.io/projected/f80c0763-ded0-4716-b424-6d10dcfa1c24-kube-api-access-4wmk2\") pod \"service-ca-568869fdc5-g7849\" (UID: \"f80c0763-ded0-4716-b424-6d10dcfa1c24\") " pod="openshift-service-ca/service-ca-568869fdc5-g7849"
Sep 15 16:27:32.083107 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:32.083073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f80c0763-ded0-4716-b424-6d10dcfa1c24-signing-cabundle\") pod \"service-ca-568869fdc5-g7849\" (UID: \"f80c0763-ded0-4716-b424-6d10dcfa1c24\") " pod="openshift-service-ca/service-ca-568869fdc5-g7849"
Sep 15 16:27:32.083689 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:32.083669    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f80c0763-ded0-4716-b424-6d10dcfa1c24-signing-cabundle\") pod \"service-ca-568869fdc5-g7849\" (UID: \"f80c0763-ded0-4716-b424-6d10dcfa1c24\") " pod="openshift-service-ca/service-ca-568869fdc5-g7849"
Sep 15 16:27:32.085271 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:32.085256    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f80c0763-ded0-4716-b424-6d10dcfa1c24-signing-key\") pod \"service-ca-568869fdc5-g7849\" (UID: \"f80c0763-ded0-4716-b424-6d10dcfa1c24\") " pod="openshift-service-ca/service-ca-568869fdc5-g7849"
Sep 15 16:27:32.092271 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:32.092243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wmk2\" (UniqueName: \"kubernetes.io/projected/f80c0763-ded0-4716-b424-6d10dcfa1c24-kube-api-access-4wmk2\") pod \"service-ca-568869fdc5-g7849\" (UID: \"f80c0763-ded0-4716-b424-6d10dcfa1c24\") " pod="openshift-service-ca/service-ca-568869fdc5-g7849"
Sep 15 16:27:32.179381 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:32.179350    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-g7849"
Sep 15 16:27:32.183722 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:32.183680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:32.183857 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:32.183812    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 16:27:32.183913 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:32.183870    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls podName:ef4b4ee7-812d-4ed4-9225-c2f5616c8133 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:40.183856037 +0000 UTC m=+51.228241752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls") pod "dns-default-czx4f" (UID: "ef4b4ee7-812d-4ed4-9225-c2f5616c8133") : secret "dns-default-metrics-tls" not found
Sep 15 16:27:32.729094 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:32.728917    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-g7849"]
Sep 15 16:27:32.732917 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:32.732898    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7np85_5a34bc31-c8e6-4214-9762-366d6fe23190/node-ca/0.log"
Sep 15 16:27:32.778659 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:27:32.778628    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80c0763_ded0_4716_b424_6d10dcfa1c24.slice/crio-fc3f2108048cf73ea63427a7c9ca7979d92b27928dd911b4d0459d8e73484c93 WatchSource:0}: Error finding container fc3f2108048cf73ea63427a7c9ca7979d92b27928dd911b4d0459d8e73484c93: Status 404 returned error can't find the container with id fc3f2108048cf73ea63427a7c9ca7979d92b27928dd911b4d0459d8e73484c93
Sep 15 16:27:32.994883 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:32.994860    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:33.595101 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:33.595056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/148e3028-957d-460a-abfe-e936406d0989-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:33.595266 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:33.595158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:33.595307 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:33.595283    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:33.595381 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:33.595368    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls podName:148e3028-957d-460a-abfe-e936406d0989 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:37.595346703 +0000 UTC m=+48.639732429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwbnk" (UID: "148e3028-957d-460a-abfe-e936406d0989") : secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:33.595725 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:33.595683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/148e3028-957d-460a-abfe-e936406d0989-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:33.640161 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:33.640130    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:33.691724 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:33.691680    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-d4pf4" event={"ID":"2bd47ace-dade-4b8d-af86-a27937a4ac38","Type":"ContainerStarted","Data":"4756e7d5d98623f6dd35b6f902b9a3b7393b6b608dd7b64a834f4c80f9da8e29"}
Sep 15 16:27:33.691851 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:33.691752    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-d4pf4" event={"ID":"2bd47ace-dade-4b8d-af86-a27937a4ac38","Type":"ContainerStarted","Data":"743c21d9c01f0e0d413990564d71bb314062411ff23cd00a3c868a312eaf00f7"}
Sep 15 16:27:33.692923 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:33.692899    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-g7849" event={"ID":"f80c0763-ded0-4716-b424-6d10dcfa1c24","Type":"ContainerStarted","Data":"fc3f2108048cf73ea63427a7c9ca7979d92b27928dd911b4d0459d8e73484c93"}
Sep 15 16:27:33.715715 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:33.715671    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-d4pf4" podStartSLOduration=2.12307503 podStartE2EDuration="3.715658515s" totalImagesPullingTime="1.592583485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:33.71504734 +0000 UTC m=+44.759433074" watchObservedRunningTime="2026-09-15 16:27:33.715658515 +0000 UTC m=+44.760044248"
Sep 15 16:27:33.725893 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:33.725846    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kdqvv_aaa40fc5-09b9-4768-95b9-005e46e994ed/serve-healthcheck-canary/0.log"
Sep 15 16:27:36.703174 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:36.703144    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-g7849" event={"ID":"f80c0763-ded0-4716-b424-6d10dcfa1c24","Type":"ContainerStarted","Data":"a7d7019cd77f2488f53d5cfa3edf4a39c92662e478ca8858c85f40f18639e535"}
Sep 15 16:27:36.729645 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:36.729590    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-g7849" podStartSLOduration=2.305602122 podStartE2EDuration="5.729575654s" totalImagesPullingTime="3.423973532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:36.728917451 +0000 UTC m=+47.773303185" watchObservedRunningTime="2026-09-15 16:27:36.729575654 +0000 UTC m=+47.773961385"
Sep 15 16:27:37.256884 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:37.256845    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:37.497764 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:37.497692    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:37.607438 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:37.607318    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:37.629025 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:37.628982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:37.629215 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:37.629151    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:37.629284 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:37.629230    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls podName:148e3028-957d-460a-abfe-e936406d0989 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:45.629208705 +0000 UTC m=+56.673594430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwbnk" (UID: "148e3028-957d-460a-abfe-e936406d0989") : secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:37.707700 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:37.707665    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:38.938772 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:38.938704    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:39.033149 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:39.033092    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:39.527328 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:39.527285    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:39.826749 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:39.826659    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:39.961575 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:39.961537    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:40.116168 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:40.116085    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:40.207457 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:40.207396    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:40.248406 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:40.248374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:40.248578 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:40.248545    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 16:27:40.248659 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:27:40.248646    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls podName:ef4b4ee7-812d-4ed4-9225-c2f5616c8133 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:56.248621674 +0000 UTC m=+67.293007386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls") pod "dns-default-czx4f" (UID: "ef4b4ee7-812d-4ed4-9225-c2f5616c8133") : secret "dns-default-metrics-tls" not found
Sep 15 16:27:41.145701 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:41.145665    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:41.248988 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:41.248959    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:41.356943 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:41.356908    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:41.539019 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:41.538938    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:41.731074 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:41.731041    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:41.931547 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:41.931512    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:42.031999 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:42.031971    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:42.133913 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:42.133876    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:43.161578 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:43.161544    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:43.250649 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:43.250617    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:43.347218 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:43.347183    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:43.533447 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:43.533298    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:43.690578 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:43.690537    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:43.794568 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:43.794483    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:43.958617 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:43.958578    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:44.157685 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:44.157647    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:45.161018 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:45.160980    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:45.540126 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:45.540038    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:45.688724 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:45.688675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:45.690970 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:45.690947    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/148e3028-957d-460a-abfe-e936406d0989-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwbnk\" (UID: \"148e3028-957d-460a-abfe-e936406d0989\") " pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:45.711008 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:45.710984    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mwbnk"
Sep 15 16:27:45.740852 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:45.740805    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:45.924920 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:45.924892    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:45.982030 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:45.982003    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mwbnk"]
Sep 15 16:27:45.984527 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:27:45.984501    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148e3028_957d_460a_abfe_e936406d0989.slice/crio-5e8bdb44e950c46d8e7714defe0f61eeab686a39694fbae7a8cdb98f715f2b50 WatchSource:0}: Error finding container 5e8bdb44e950c46d8e7714defe0f61eeab686a39694fbae7a8cdb98f715f2b50: Status 404 returned error can't find the container with id 5e8bdb44e950c46d8e7714defe0f61eeab686a39694fbae7a8cdb98f715f2b50
Sep 15 16:27:46.029174 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:46.029147    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:46.370956 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:46.370869    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:46.731653 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:46.731624    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwbnk" event={"ID":"148e3028-957d-460a-abfe-e936406d0989","Type":"ContainerStarted","Data":"3aa6f8e800f89b6143d3a24cacf35001b2b485d67c69424e9a6f4d2c07553884"}
Sep 15 16:27:46.731816 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:46.731676    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwbnk" event={"ID":"148e3028-957d-460a-abfe-e936406d0989","Type":"ContainerStarted","Data":"7c0eef580350fb65be37fe036dee028b8cb47efdff3ac08b1aa6e264cc20b698"}
Sep 15 16:27:46.731816 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:46.731697    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwbnk" event={"ID":"148e3028-957d-460a-abfe-e936406d0989","Type":"ContainerStarted","Data":"5e8bdb44e950c46d8e7714defe0f61eeab686a39694fbae7a8cdb98f715f2b50"}
Sep 15 16:27:46.935248 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:46.935215    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:47.532118 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:47.532078    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:47.725098 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:47.725062    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:47.924852 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:47.924804    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:48.126681 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:48.126647    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:48.653485 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:48.653452    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9bwpd"
Sep 15 16:27:48.739645 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:48.739613    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwbnk" event={"ID":"148e3028-957d-460a-abfe-e936406d0989","Type":"ContainerStarted","Data":"3ad6468ce2b38b59df68384df529791e5ddd7d6958c1e1f624ec848565139749"}
Sep 15 16:27:48.761418 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:48.761370    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mwbnk" podStartSLOduration=17.505781838 podStartE2EDuration="19.761354553s" totalImagesPullingTime="2.255572715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:48.760046993 +0000 UTC m=+59.804432726" watchObservedRunningTime="2026-09-15 16:27:48.761354553 +0000 UTC m=+59.805740279"
Sep 15 16:27:53.289474 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.289432    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"]
Sep 15 16:27:53.293914 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.293890    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"]
Sep 15 16:27:53.294048 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.294008    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"
Sep 15 16:27:53.296420 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.296402    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 16:27:53.296518 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.296404    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 15 16:27:53.296668 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.296652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.296849 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.296830    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 16:27:53.297138 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.297123    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 16:27:53.299159 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.299135    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 15 16:27:53.299556 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.299497    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 15 16:27:53.299733 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.299719    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 15 16:27:53.299880 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.299863    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 15 16:27:53.304333 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.304309    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"]
Sep 15 16:27:53.305190 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.305172    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"]
Sep 15 16:27:53.313568 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.313544    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:53.341676 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.341652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clmr7\" (UniqueName: \"kubernetes.io/projected/2d6cf578-85fe-436b-8591-9920c3a70926-kube-api-access-clmr7\") pod \"klusterlet-addon-workmgr-595844db5b-hhk68\" (UID: \"2d6cf578-85fe-436b-8591-9920c3a70926\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"
Sep 15 16:27:53.341784 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.341681    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0726a527-379a-4c36-8e3a-9028fa99a533-hub\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.341784 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.341696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0726a527-379a-4c36-8e3a-9028fa99a533-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.341854 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.341799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0726a527-379a-4c36-8e3a-9028fa99a533-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.341854 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.341843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0726a527-379a-4c36-8e3a-9028fa99a533-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.341920 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.341860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vntrw\" (UniqueName: \"kubernetes.io/projected/0726a527-379a-4c36-8e3a-9028fa99a533-kube-api-access-vntrw\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.341920 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.341903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2d6cf578-85fe-436b-8591-9920c3a70926-klusterlet-config\") pod \"klusterlet-addon-workmgr-595844db5b-hhk68\" (UID: \"2d6cf578-85fe-436b-8591-9920c3a70926\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"
Sep 15 16:27:53.341980 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.341920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0726a527-379a-4c36-8e3a-9028fa99a533-ca\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.342019 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.342006    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d6cf578-85fe-436b-8591-9920c3a70926-tmp\") pod \"klusterlet-addon-workmgr-595844db5b-hhk68\" (UID: \"2d6cf578-85fe-436b-8591-9920c3a70926\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"
Sep 15 16:27:53.439324 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.439291    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:53.442284 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.442262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0726a527-379a-4c36-8e3a-9028fa99a533-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.442414 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.442295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0726a527-379a-4c36-8e3a-9028fa99a533-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.442414 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.442320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntrw\" (UniqueName: \"kubernetes.io/projected/0726a527-379a-4c36-8e3a-9028fa99a533-kube-api-access-vntrw\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.442414 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.442365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2d6cf578-85fe-436b-8591-9920c3a70926-klusterlet-config\") pod \"klusterlet-addon-workmgr-595844db5b-hhk68\" (UID: \"2d6cf578-85fe-436b-8591-9920c3a70926\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"
Sep 15 16:27:53.442414 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.442388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0726a527-379a-4c36-8e3a-9028fa99a533-ca\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.442619 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.442469    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d6cf578-85fe-436b-8591-9920c3a70926-tmp\") pod \"klusterlet-addon-workmgr-595844db5b-hhk68\" (UID: \"2d6cf578-85fe-436b-8591-9920c3a70926\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"
Sep 15 16:27:53.442619 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.442507    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clmr7\" (UniqueName: \"kubernetes.io/projected/2d6cf578-85fe-436b-8591-9920c3a70926-kube-api-access-clmr7\") pod \"klusterlet-addon-workmgr-595844db5b-hhk68\" (UID: \"2d6cf578-85fe-436b-8591-9920c3a70926\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"
Sep 15 16:27:53.442619 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.442531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0726a527-379a-4c36-8e3a-9028fa99a533-hub\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.442619 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.442554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0726a527-379a-4c36-8e3a-9028fa99a533-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.442981 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.442948    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d6cf578-85fe-436b-8591-9920c3a70926-tmp\") pod \"klusterlet-addon-workmgr-595844db5b-hhk68\" (UID: \"2d6cf578-85fe-436b-8591-9920c3a70926\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"
Sep 15 16:27:53.443128 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.443058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0726a527-379a-4c36-8e3a-9028fa99a533-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.445122 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.445092    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0726a527-379a-4c36-8e3a-9028fa99a533-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.445229 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.445188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0726a527-379a-4c36-8e3a-9028fa99a533-hub\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.445299 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.445259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0726a527-379a-4c36-8e3a-9028fa99a533-ca\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.445380 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.445358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2d6cf578-85fe-436b-8591-9920c3a70926-klusterlet-config\") pod \"klusterlet-addon-workmgr-595844db5b-hhk68\" (UID: \"2d6cf578-85fe-436b-8591-9920c3a70926\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"
Sep 15 16:27:53.445480 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.445463    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0726a527-379a-4c36-8e3a-9028fa99a533-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.453689 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.453666    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntrw\" (UniqueName: \"kubernetes.io/projected/0726a527-379a-4c36-8e3a-9028fa99a533-kube-api-access-vntrw\") pod \"cluster-proxy-proxy-agent-75674888f6-j54vc\" (UID: \"0726a527-379a-4c36-8e3a-9028fa99a533\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.459335 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.459313    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clmr7\" (UniqueName: \"kubernetes.io/projected/2d6cf578-85fe-436b-8591-9920c3a70926-kube-api-access-clmr7\") pod \"klusterlet-addon-workmgr-595844db5b-hhk68\" (UID: \"2d6cf578-85fe-436b-8591-9920c3a70926\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"
Sep 15 16:27:53.541805 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.541720    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:53.604021 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.603992    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"
Sep 15 16:27:53.618160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.618129    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"
Sep 15 16:27:53.643336 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.643292    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:53.740978 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.740941    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:53.888618 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.888533    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"]
Sep 15 16:27:53.893100 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:27:53.893074    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6cf578_85fe_436b_8591_9920c3a70926.slice/crio-b31b8be58c2366058849ab193dab92780dad65655e3ed7b1b12514576d471f5b WatchSource:0}: Error finding container b31b8be58c2366058849ab193dab92780dad65655e3ed7b1b12514576d471f5b: Status 404 returned error can't find the container with id b31b8be58c2366058849ab193dab92780dad65655e3ed7b1b12514576d471f5b
Sep 15 16:27:53.910009 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:53.909971    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc"]
Sep 15 16:27:53.912466 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:27:53.912446    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0726a527_379a_4c36_8e3a_9028fa99a533.slice/crio-0523f58ca4a7f31aca27e118c091c0f5fc6bc671c523851d3bd6ce74942d99c1 WatchSource:0}: Error finding container 0523f58ca4a7f31aca27e118c091c0f5fc6bc671c523851d3bd6ce74942d99c1: Status 404 returned error can't find the container with id 0523f58ca4a7f31aca27e118c091c0f5fc6bc671c523851d3bd6ce74942d99c1
Sep 15 16:27:54.765636 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:54.765582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc" event={"ID":"0726a527-379a-4c36-8e3a-9028fa99a533","Type":"ContainerStarted","Data":"0523f58ca4a7f31aca27e118c091c0f5fc6bc671c523851d3bd6ce74942d99c1"}
Sep 15 16:27:54.767480 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:54.767393    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68" event={"ID":"2d6cf578-85fe-436b-8591-9920c3a70926","Type":"ContainerStarted","Data":"b31b8be58c2366058849ab193dab92780dad65655e3ed7b1b12514576d471f5b"}
Sep 15 16:27:55.131249 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.131168    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:55.155806 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.155773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs\") pod \"network-metrics-daemon-nvl8x\" (UID: \"9a34ad6e-33f8-4f96-9aca-98998f3bcd62\") " pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:55.159038 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.158931    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 16:27:55.169511 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.169490    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a34ad6e-33f8-4f96-9aca-98998f3bcd62-metrics-certs\") pod \"network-metrics-daemon-nvl8x\" (UID: \"9a34ad6e-33f8-4f96-9aca-98998f3bcd62\") " pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:55.236998 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.236885    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:55.256603 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.256526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:55.259518 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.259287    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 16:27:55.278999 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.278970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22495e12-7c88-4178-9d83-076062a2c424-original-pull-secret\") pod \"global-pull-secret-syncer-jvr5t\" (UID: \"22495e12-7c88-4178-9d83-076062a2c424\") " pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:55.322723 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.322667    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:55.357307 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.357272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4nr\" (UniqueName: \"kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr\") pod \"network-check-target-vjsq5\" (UID: \"33198472-1392-416c-a639-0ba97f72fd7f\") " pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:55.361022 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.360754    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 16:27:55.371278 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.371256    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 16:27:55.383971 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.383910    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb4nr\" (UniqueName: \"kubernetes.io/projected/33198472-1392-416c-a639-0ba97f72fd7f-kube-api-access-sb4nr\") pod \"network-check-target-vjsq5\" (UID: \"33198472-1392-416c-a639-0ba97f72fd7f\") " pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:55.433597 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.433568    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:55.456275 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.455940    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9w9qq\""
Sep 15 16:27:55.461679 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.461654    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vcckz\""
Sep 15 16:27:55.463202 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.463182    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jvr5t"
Sep 15 16:27:55.463449 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.463432    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nvl8x"
Sep 15 16:27:55.469394 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.469369    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:27:55.524028 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.523996    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:55.790786 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.790522    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jvr5t"]
Sep 15 16:27:55.807063 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.806977    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:55.834329 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.834245    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nvl8x"]
Sep 15 16:27:55.854451 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:55.854408    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vjsq5"]
Sep 15 16:27:56.265752 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:56.265696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:56.269483 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:56.269453    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef4b4ee7-812d-4ed4-9225-c2f5616c8133-metrics-tls\") pod \"dns-default-czx4f\" (UID: \"ef4b4ee7-812d-4ed4-9225-c2f5616c8133\") " pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:56.479285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:56.479259    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xg4rz\""
Sep 15 16:27:56.488097 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:56.488049    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-czx4f"
Sep 15 16:27:56.778554 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:56.778505    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nvl8x" event={"ID":"9a34ad6e-33f8-4f96-9aca-98998f3bcd62","Type":"ContainerStarted","Data":"0401ae697321d1694acd527cd0b831a04c848f059cfe7dcdeb653cc87412622b"}
Sep 15 16:27:56.780774 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:56.780654    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vjsq5" event={"ID":"33198472-1392-416c-a639-0ba97f72fd7f","Type":"ContainerStarted","Data":"3687ea8f2edda51f4623d03f9a8d734d0f38a0523a8c7511868d7713a03b615a"}
Sep 15 16:27:56.782900 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:56.782836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jvr5t" event={"ID":"22495e12-7c88-4178-9d83-076062a2c424","Type":"ContainerStarted","Data":"918a61488ecb8cfeafa2ff52ca3b528093e90347280a477532a97cd2586c4748"}
Sep 15 16:27:57.772667 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:57.772607    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:27:57.838608 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:27:57.838560    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:01.270403 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:01.270361    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:01.340274 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:01.340238    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:01.705724 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:01.705677    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:02.185571 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:02.185532    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-czx4f"]
Sep 15 16:28:02.901640 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:28:02.901613    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4b4ee7_812d_4ed4_9225_c2f5616c8133.slice/crio-d5e11360ed9c7feb9d5b74bf59bbe4fd4f91013ae1c3b6584efada1772de67b9 WatchSource:0}: Error finding container d5e11360ed9c7feb9d5b74bf59bbe4fd4f91013ae1c3b6584efada1772de67b9: Status 404 returned error can't find the container with id d5e11360ed9c7feb9d5b74bf59bbe4fd4f91013ae1c3b6584efada1772de67b9
Sep 15 16:28:03.810140 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.810101    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc" event={"ID":"0726a527-379a-4c36-8e3a-9028fa99a533","Type":"ContainerStarted","Data":"2bcef8fa5098d3fd8840df277ccff2fd1bc940b60d7be7b32f08c3f8aa1bd000"}
Sep 15 16:28:03.810483 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.810154    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc" event={"ID":"0726a527-379a-4c36-8e3a-9028fa99a533","Type":"ContainerStarted","Data":"8ccf662229d5776d3eb67c15741ca89972feb664b308ab6dca5a74dba43ae477"}
Sep 15 16:28:03.810483 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.810176    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc" event={"ID":"0726a527-379a-4c36-8e3a-9028fa99a533","Type":"ContainerStarted","Data":"ed3701da10901684319836f181cf85c41985683b03055326a9adfeabf5a16484"}
Sep 15 16:28:03.811474 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.811450    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68" event={"ID":"2d6cf578-85fe-436b-8591-9920c3a70926","Type":"ContainerStarted","Data":"99bc3a6f8ba319f15e523550d577924d5bfa7807af9078b804d2c843f1c619f9"}
Sep 15 16:28:03.811554 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.811497    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"
Sep 15 16:28:03.813293 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.813266    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nvl8x" event={"ID":"9a34ad6e-33f8-4f96-9aca-98998f3bcd62","Type":"ContainerStarted","Data":"a4d27a3f2a8a7ed568bf8d0e5c66eaf733634bc1900c7a9f9bb818101c2028ff"}
Sep 15 16:28:03.813430 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.813314    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nvl8x" event={"ID":"9a34ad6e-33f8-4f96-9aca-98998f3bcd62","Type":"ContainerStarted","Data":"1fc969c68ef013c592dfd96c8e2f585153b742ce477b5336787f36dfeb32cedf"}
Sep 15 16:28:03.813490 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.813429    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68"
Sep 15 16:28:03.814544 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.814510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-czx4f" event={"ID":"ef4b4ee7-812d-4ed4-9225-c2f5616c8133","Type":"ContainerStarted","Data":"d5e11360ed9c7feb9d5b74bf59bbe4fd4f91013ae1c3b6584efada1772de67b9"}
Sep 15 16:28:03.815993 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.815974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vjsq5" event={"ID":"33198472-1392-416c-a639-0ba97f72fd7f","Type":"ContainerStarted","Data":"807abe51442a539fb405c258d92a8d3a07fe7930a374443864189042a97ded43"}
Sep 15 16:28:03.816102 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.816027    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:28:03.817389 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.817361    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jvr5t" event={"ID":"22495e12-7c88-4178-9d83-076062a2c424","Type":"ContainerStarted","Data":"91a5fceb546a7673b5b2981a7cb31a022e62428fb6b83b137a23b7439c257bd4"}
Sep 15 16:28:03.832456 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.832408    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75674888f6-j54vc" podStartSLOduration=1.8469839559999999 podStartE2EDuration="10.832389982s" totalImagesPullingTime="8.985406026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:03.830340847 +0000 UTC m=+74.874726592" watchObservedRunningTime="2026-09-15 16:28:03.832389982 +0000 UTC m=+74.876775713"
Sep 15 16:28:03.851687 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.851647    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nvl8x" podStartSLOduration=68.667302999 podStartE2EDuration="1m14.851632597s" totalImagesPullingTime="6.184329598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:03.850976386 +0000 UTC m=+74.895362119" watchObservedRunningTime="2026-09-15 16:28:03.851632597 +0000 UTC m=+74.896018368"
Sep 15 16:28:03.890570 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.890522    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vjsq5" podStartSLOduration=67.668403228 podStartE2EDuration="1m13.89050753s" totalImagesPullingTime="6.222104302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:03.871668598 +0000 UTC m=+74.916054341" watchObservedRunningTime="2026-09-15 16:28:03.89050753 +0000 UTC m=+74.934893262"
Sep 15 16:28:03.908433 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.908380    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jvr5t" podStartSLOduration=67.6997402 podStartE2EDuration="1m13.908365446s" totalImagesPullingTime="6.208625246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:03.890096416 +0000 UTC m=+74.934482148" watchObservedRunningTime="2026-09-15 16:28:03.908365446 +0000 UTC m=+74.952751176"
Sep 15 16:28:03.909650 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.909610    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-595844db5b-hhk68" podStartSLOduration=1.9048090050000002 podStartE2EDuration="10.909598306s" totalImagesPullingTime="9.004789301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:03.908355877 +0000 UTC m=+74.952741612" watchObservedRunningTime="2026-09-15 16:28:03.909598306 +0000 UTC m=+74.953984039"
Sep 15 16:28:03.949533 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:03.949506    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:04.048422 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:04.048381    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:04.102521 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:04.102428    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:04.759636 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:04.759550    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:04.822268 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:04.822229    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-czx4f" event={"ID":"ef4b4ee7-812d-4ed4-9225-c2f5616c8133","Type":"ContainerStarted","Data":"3f928a9583d9507f9303849e70c701a2d9498f20659366b0feec2e4eed33f355"}
Sep 15 16:28:04.822435 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:04.822325    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-czx4f" event={"ID":"ef4b4ee7-812d-4ed4-9225-c2f5616c8133","Type":"ContainerStarted","Data":"d202617cadc59200c6dd3f5e6199482173a986cb75f36edcee86269c343537e7"}
Sep 15 16:28:04.844686 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:04.844631    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-czx4f" podStartSLOduration=39.258852809 podStartE2EDuration="40.844614309s" totalImagesPullingTime="1.5857615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:04.843246188 +0000 UTC m=+75.887631919" watchObservedRunningTime="2026-09-15 16:28:04.844614309 +0000 UTC m=+75.889000040"
Sep 15 16:28:05.826523 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:05.826491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-czx4f"
Sep 15 16:28:06.532360 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.532325    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:06.648444 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.648415    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:06.874062 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.873988    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gjj84"]
Sep 15 16:28:06.879286 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.879268    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:06.885338 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.885301    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 16:28:06.886351 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.886327    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 16:28:06.886477 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.886365    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 16:28:06.886477 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.886331    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 16:28:06.886477 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.886329    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cj2v6\""
Sep 15 16:28:06.886477 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.886338    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 16:28:06.886477 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.886343    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 16:28:06.943627 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.943595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2e585b9-a509-4241-b3f0-7432f5633b21-root\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:06.943627 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.943628    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2e585b9-a509-4241-b3f0-7432f5633b21-metrics-client-ca\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:06.943820 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.943651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2e585b9-a509-4241-b3f0-7432f5633b21-sys\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:06.943820 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.943767    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:06.943820 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.943797    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-wtmp\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:06.943820 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.943820    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-textfile\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:06.943950 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.943843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-schcc\" (UniqueName: \"kubernetes.io/projected/e2e585b9-a509-4241-b3f0-7432f5633b21-kube-api-access-schcc\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:06.943950 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.943907    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-accelerators-collector-config\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:06.944013 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:06.943955    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-tls\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045013 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.044976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-accelerators-collector-config\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045130 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-tls\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045130 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2e585b9-a509-4241-b3f0-7432f5633b21-root\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045130 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2e585b9-a509-4241-b3f0-7432f5633b21-metrics-client-ca\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045130 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2e585b9-a509-4241-b3f0-7432f5633b21-sys\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045295 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045185    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2e585b9-a509-4241-b3f0-7432f5633b21-sys\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045295 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045193    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2e585b9-a509-4241-b3f0-7432f5633b21-root\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045295 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045248    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045401 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-wtmp\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045401 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-textfile\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045401 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-schcc\" (UniqueName: \"kubernetes.io/projected/e2e585b9-a509-4241-b3f0-7432f5633b21-kube-api-access-schcc\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045536 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-wtmp\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045724 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045692    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-accelerators-collector-config\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045819 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-textfile\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.045819 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.045692    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2e585b9-a509-4241-b3f0-7432f5633b21-metrics-client-ca\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.047484 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.047463    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-tls\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.047580 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.047494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2e585b9-a509-4241-b3f0-7432f5633b21-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.054476 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.054452    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-schcc\" (UniqueName: \"kubernetes.io/projected/e2e585b9-a509-4241-b3f0-7432f5633b21-kube-api-access-schcc\") pod \"node-exporter-gjj84\" (UID: \"e2e585b9-a509-4241-b3f0-7432f5633b21\") " pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.189003 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.188957    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gjj84"
Sep 15 16:28:07.202601 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:28:07.202575    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e585b9_a509_4241_b3f0_7432f5633b21.slice/crio-81292b62ab38a29fe28434bcf59b7f2c9c026b42e0d6b0419aaa6706aae2738e WatchSource:0}: Error finding container 81292b62ab38a29fe28434bcf59b7f2c9c026b42e0d6b0419aaa6706aae2738e: Status 404 returned error can't find the container with id 81292b62ab38a29fe28434bcf59b7f2c9c026b42e0d6b0419aaa6706aae2738e
Sep 15 16:28:07.265609 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.265582    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:07.363997 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.363965    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:07.473204 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.473120    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:07.781864 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.781779    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:07.835961 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.835925    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gjj84" event={"ID":"e2e585b9-a509-4241-b3f0-7432f5633b21","Type":"ContainerStarted","Data":"81292b62ab38a29fe28434bcf59b7f2c9c026b42e0d6b0419aaa6706aae2738e"}
Sep 15 16:28:07.863647 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:07.863613    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:08.749404 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:08.749371    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:08.842901 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:08.842867    2565 generic.go:356] "Generic (PLEG): container finished" podID="e2e585b9-a509-4241-b3f0-7432f5633b21" containerID="9515283053e41b1b9c11f00c4182152b273a0fd9f5c61cb7924b50110b80dd80" exitCode=0
Sep 15 16:28:08.842901 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:08.842907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gjj84" event={"ID":"e2e585b9-a509-4241-b3f0-7432f5633b21","Type":"ContainerDied","Data":"9515283053e41b1b9c11f00c4182152b273a0fd9f5c61cb7924b50110b80dd80"}
Sep 15 16:28:09.706532 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:09.706498    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:09.807374 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:09.807345    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:09.847888 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:09.847856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gjj84" event={"ID":"e2e585b9-a509-4241-b3f0-7432f5633b21","Type":"ContainerStarted","Data":"6d7f4afe9d082f4268a3015292a9655cd5dff1e9b56af42c5b9385cbf812b841"}
Sep 15 16:28:09.848035 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:09.847904    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gjj84" event={"ID":"e2e585b9-a509-4241-b3f0-7432f5633b21","Type":"ContainerStarted","Data":"8fa99280c8ff655b78ce015abd76378d0d4a1e0559a0d7938fbfd4b027ea7b47"}
Sep 15 16:28:09.871504 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:09.871462    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gjj84" podStartSLOduration=2.627335366 podStartE2EDuration="3.87144783s" totalImagesPullingTime="1.181248375s" totalInitContainerRuntime="62.864089ms" isStatefulPod=true podCreationTimestamp="2026-09-15 16:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:09.869791494 +0000 UTC m=+80.914177291" watchObservedRunningTime="2026-09-15 16:28:09.87144783 +0000 UTC m=+80.915833566"
Sep 15 16:28:09.874548 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:09.874528    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:10.040097 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:10.040017    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:10.140514 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:10.140486    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:11.451655 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:11.451621    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:11.772434 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:11.772351    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:13.030473 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:13.030435    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:13.488381 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:13.488341    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:13.869055 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:13.868977    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:14.469053 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:14.469019    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:14.533359 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:14.533326    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:15.790866 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:15.790833    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:15.833929 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:15.833902    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-czx4f"
Sep 15 16:28:15.907396 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:15.907365    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:16.006959 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:16.006926    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:18.180947 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:18.180909    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:18.282195 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:18.282156    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:22.224150 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:22.224117    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:22.323266 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:22.323229    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:22.572678 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:22.572603    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:22.834675 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:22.834570    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:22.935466 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:22.935432    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:23.036132 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:23.036095    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:23.243095 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:23.243063    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:23.341363 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:23.341334    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:23.442622 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:23.442591    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:23.635042 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:23.634961    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:23.735170 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:23.735140    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:24.274260 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:24.274225    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:24.375447 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:24.375412    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:24.475722 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:24.475671    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:24.635973 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:24.635887    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:24.907172 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:24.907137    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:25.025942 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:25.025911    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:25.126627 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:25.126592    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:27.300722 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:27.300678    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:27.381441 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:27.381365    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:32.394873 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:32.394833    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:32.456725 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:32.456675    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:34.825177 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:34.825146    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vjsq5"
Sep 15 16:28:35.066664 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:35.066628    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:35.130570 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:35.130500    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:38.861292 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:38.861253    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:39.065197 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:39.065162    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:40.025509 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:40.025476    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:40.085738 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:40.085698    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:47.457942 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:47.457899    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:47.522078 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:47.522046    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:48.283579 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:48.283535    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:48.341769 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:48.341696    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:49.688292 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:49.688234    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:49.788537 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:49.788485    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:49.851854 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:49.851804    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:50.697173 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:50.697142    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:50.751521 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:50.751492    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:51.284921 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:51.284887    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:51.380312 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:51.380283    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:55.022850 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:55.022819    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:28:55.123344 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:28:55.123309    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:29:10.033559 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:29:10.033525    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:29:10.090606 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:29:10.090568    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:29:26.002752 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:29:26.002697    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:29:26.206222 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:29:26.206191    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:29:33.871752 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:29:33.871698    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:29:33.928669 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:29:33.928637    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:29:45.354262 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:29:45.354175    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:29:45.410652 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:29:45.410619    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:30:41.216856 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:30:41.216816    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:30:41.268872 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:30:41.268838    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:30:48.063436 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:30:48.063389    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:30:48.131982 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:30:48.131957    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:30:53.217957 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:30:53.217927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:30:53.319149 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:30:53.319103    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:30:59.229686 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:30:59.229652    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:30:59.327607 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:30:59.327572    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:30:59.386982 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:30:59.386955    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:31:04.465345 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:31:04.465307    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:31:10.059408 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:31:10.059371    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:31:10.157796 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:31:10.157766    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:31:16.389240 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:31:16.389152    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:31:49.366822 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:31:49.366783    2565 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 16:32:23.640012 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:32:23.639980    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:32:27.078504 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:32:27.078475    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:32:40.493217 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:32:40.493164    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:33:37.625627 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:33:37.625585    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:33:37.683701 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:33:37.683670    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:33:44.115949 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:33:44.115909    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:33:44.213330 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:33:44.213293    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:33:44.268249 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:33:44.268222    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:33:48.926190 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:33:48.926157    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:33:49.026954 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:33:49.026920    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:33:49.086233 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:33:49.086200    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:11.732820 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:11.732782    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:11.840123 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:11.840089    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:17.017901 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.017864    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:17.304151 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.304048    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:17.309393 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.309366    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-b7vp4"]
Sep 15 16:34:17.311088 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.311072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:17.313795 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.313775    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 15 16:34:17.313991 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.313975    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 15 16:34:17.314819 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.314793    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-hh427\""
Sep 15 16:34:17.314819 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.314799    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 15 16:34:17.314968 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.314829    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 15 16:34:17.314968 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.314827    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 15 16:34:17.326402 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.326368    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-b7vp4"]
Sep 15 16:34:17.409677 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.409639    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-cabundle0\") pod \"keda-operator-579c8958f7-b7vp4\" (UID: \"ee66e697-ea8e-4ced-a37c-c23cf109d4fc\") " pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:17.409858 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.409697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-certificates\") pod \"keda-operator-579c8958f7-b7vp4\" (UID: \"ee66e697-ea8e-4ced-a37c-c23cf109d4fc\") " pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:17.409858 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.409831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prfzl\" (UniqueName: \"kubernetes.io/projected/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-kube-api-access-prfzl\") pod \"keda-operator-579c8958f7-b7vp4\" (UID: \"ee66e697-ea8e-4ced-a37c-c23cf109d4fc\") " pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:17.496829 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.496797    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:17.510420 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.510393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-certificates\") pod \"keda-operator-579c8958f7-b7vp4\" (UID: \"ee66e697-ea8e-4ced-a37c-c23cf109d4fc\") " pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:17.510582 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.510435    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prfzl\" (UniqueName: \"kubernetes.io/projected/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-kube-api-access-prfzl\") pod \"keda-operator-579c8958f7-b7vp4\" (UID: \"ee66e697-ea8e-4ced-a37c-c23cf109d4fc\") " pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:17.510582 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.510541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-cabundle0\") pod \"keda-operator-579c8958f7-b7vp4\" (UID: \"ee66e697-ea8e-4ced-a37c-c23cf109d4fc\") " pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:17.510582 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:34:17.510554    2565 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 15 16:34:17.510582 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:34:17.510575    2565 secret.go:281] references non-existent secret key: ca.crt
Sep 15 16:34:17.510582 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:34:17.510585    2565 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 15 16:34:17.510839 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:34:17.510600    2565 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-b7vp4: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 15 16:34:17.510839 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:34:17.510670    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-certificates podName:ee66e697-ea8e-4ced-a37c-c23cf109d4fc nodeName:}" failed. No retries permitted until 2026-09-15 16:34:18.01064611 +0000 UTC m=+449.055031822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-certificates") pod "keda-operator-579c8958f7-b7vp4" (UID: "ee66e697-ea8e-4ced-a37c-c23cf109d4fc") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 15 16:34:17.511157 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.511139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-cabundle0\") pod \"keda-operator-579c8958f7-b7vp4\" (UID: \"ee66e697-ea8e-4ced-a37c-c23cf109d4fc\") " pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:17.522015 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.521992    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prfzl\" (UniqueName: \"kubernetes.io/projected/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-kube-api-access-prfzl\") pod \"keda-operator-579c8958f7-b7vp4\" (UID: \"ee66e697-ea8e-4ced-a37c-c23cf109d4fc\") " pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:17.600297 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.600158    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:17.725159 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.725129    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:17.826437 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:17.826404    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:18.014962 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:18.014927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-certificates\") pod \"keda-operator-579c8958f7-b7vp4\" (UID: \"ee66e697-ea8e-4ced-a37c-c23cf109d4fc\") " pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:18.015122 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:34:18.015041    2565 secret.go:281] references non-existent secret key: ca.crt
Sep 15 16:34:18.015122 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:34:18.015055    2565 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 15 16:34:18.015122 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:34:18.015063    2565 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-b7vp4: references non-existent secret key: ca.crt
Sep 15 16:34:18.015122 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:34:18.015121    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-certificates podName:ee66e697-ea8e-4ced-a37c-c23cf109d4fc nodeName:}" failed. No retries permitted until 2026-09-15 16:34:19.015106576 +0000 UTC m=+450.059492290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-certificates") pod "keda-operator-579c8958f7-b7vp4" (UID: "ee66e697-ea8e-4ced-a37c-c23cf109d4fc") : references non-existent secret key: ca.crt
Sep 15 16:34:18.085879 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:18.085845    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:18.182289 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:18.182258    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:18.457907 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:18.457872    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:18.659506 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:18.659479    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:18.859666 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:18.859582    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:18.960213 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:18.960178    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:34:19.022961 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:19.022924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-certificates\") pod \"keda-operator-579c8958f7-b7vp4\" (UID: \"ee66e697-ea8e-4ced-a37c-c23cf109d4fc\") " pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:19.025342 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:19.025323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ee66e697-ea8e-4ced-a37c-c23cf109d4fc-certificates\") pod \"keda-operator-579c8958f7-b7vp4\" (UID: \"ee66e697-ea8e-4ced-a37c-c23cf109d4fc\") " pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:19.120935 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:19.120844    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:19.390585 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:19.390559    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-b7vp4"]
Sep 15 16:34:19.393105 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:34:19.393075    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee66e697_ea8e_4ced_a37c_c23cf109d4fc.slice/crio-2332da237353612793dc57290c1ed177ac577bca70aee655319e358fb15932b0 WatchSource:0}: Error finding container 2332da237353612793dc57290c1ed177ac577bca70aee655319e358fb15932b0: Status 404 returned error can't find the container with id 2332da237353612793dc57290c1ed177ac577bca70aee655319e358fb15932b0
Sep 15 16:34:19.394389 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:19.394369    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:34:20.035210 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:20.035173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-b7vp4" event={"ID":"ee66e697-ea8e-4ced-a37c-c23cf109d4fc","Type":"ContainerStarted","Data":"2332da237353612793dc57290c1ed177ac577bca70aee655319e358fb15932b0"}
Sep 15 16:34:23.047744 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:23.047685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-b7vp4" event={"ID":"ee66e697-ea8e-4ced-a37c-c23cf109d4fc","Type":"ContainerStarted","Data":"78da4a30d495c364e82080e39be57cbacf80c2c5255c6a743c86605e0ef094a9"}
Sep 15 16:34:23.048108 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:23.047784    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:34:23.094303 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:23.094250    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-b7vp4" podStartSLOduration=2.940801311 podStartE2EDuration="6.094231739s" totalImagesPullingTime="3.153430428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:34:23.092609953 +0000 UTC m=+454.136995698" watchObservedRunningTime="2026-09-15 16:34:23.094231739 +0000 UTC m=+454.138617471"
Sep 15 16:34:44.054053 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:34:44.054011    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-b7vp4"
Sep 15 16:35:09.798171 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:09.798137    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:35:09.853396 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:09.853364    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:35:34.922044 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:34.921994    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:35:34.978341 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:34.978307    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp"]
Sep 15 16:35:34.981317 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:34.981300    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp"
Sep 15 16:35:34.984355 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:34.984327    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 15 16:35:34.984355 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:34.984327    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 15 16:35:34.984976 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:34.984961    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 15 16:35:34.985048 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:34.984983    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-754s5\""
Sep 15 16:35:34.991450 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:34.991426    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp"]
Sep 15 16:35:35.012675 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.012647    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:35:35.030904 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.030874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4ec8569-263e-4bea-bd51-7b5df4cfde37-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-z9sdp\" (UID: \"b4ec8569-263e-4bea-bd51-7b5df4cfde37\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp"
Sep 15 16:35:35.031023 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.030915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slxgh\" (UniqueName: \"kubernetes.io/projected/b4ec8569-263e-4bea-bd51-7b5df4cfde37-kube-api-access-slxgh\") pod \"llmisvc-controller-manager-645f9c8fd6-z9sdp\" (UID: \"b4ec8569-263e-4bea-bd51-7b5df4cfde37\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp"
Sep 15 16:35:35.085933 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.085905    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:35:35.131586 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.131561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4ec8569-263e-4bea-bd51-7b5df4cfde37-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-z9sdp\" (UID: \"b4ec8569-263e-4bea-bd51-7b5df4cfde37\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp"
Sep 15 16:35:35.131741 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.131593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slxgh\" (UniqueName: \"kubernetes.io/projected/b4ec8569-263e-4bea-bd51-7b5df4cfde37-kube-api-access-slxgh\") pod \"llmisvc-controller-manager-645f9c8fd6-z9sdp\" (UID: \"b4ec8569-263e-4bea-bd51-7b5df4cfde37\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp"
Sep 15 16:35:35.134093 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.134072    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4ec8569-263e-4bea-bd51-7b5df4cfde37-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-z9sdp\" (UID: \"b4ec8569-263e-4bea-bd51-7b5df4cfde37\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp"
Sep 15 16:35:35.139354 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.139328    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slxgh\" (UniqueName: \"kubernetes.io/projected/b4ec8569-263e-4bea-bd51-7b5df4cfde37-kube-api-access-slxgh\") pod \"llmisvc-controller-manager-645f9c8fd6-z9sdp\" (UID: \"b4ec8569-263e-4bea-bd51-7b5df4cfde37\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp"
Sep 15 16:35:35.185409 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.185346    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:35:35.287222 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.287190    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:35:35.292139 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.292123    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp"
Sep 15 16:35:35.486657 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.486578    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:35:35.560216 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.560181    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp"]
Sep 15 16:35:35.562908 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:35:35.562874    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb4ec8569_263e_4bea_bd51_7b5df4cfde37.slice/crio-7113168b1857d513b891b7d5540727d8ffb0cb241ed877bf4c2b8598399bd113 WatchSource:0}: Error finding container 7113168b1857d513b891b7d5540727d8ffb0cb241ed877bf4c2b8598399bd113: Status 404 returned error can't find the container with id 7113168b1857d513b891b7d5540727d8ffb0cb241ed877bf4c2b8598399bd113
Sep 15 16:35:35.563114 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.563093    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:35:35.622050 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:35.622021    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:35:36.290309 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:36.290269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp" event={"ID":"b4ec8569-263e-4bea-bd51-7b5df4cfde37","Type":"ContainerStarted","Data":"7113168b1857d513b891b7d5540727d8ffb0cb241ed877bf4c2b8598399bd113"}
Sep 15 16:35:38.299092 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:38.299057    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp" event={"ID":"b4ec8569-263e-4bea-bd51-7b5df4cfde37","Type":"ContainerStarted","Data":"89d0fe0f8d26dccb37bab72b8a4df9a89493657b26c98fefb49de0156b6a2138"}
Sep 15 16:35:38.299452 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:38.299191    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp"
Sep 15 16:35:38.314031 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:35:38.313986    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp" podStartSLOduration=2.003388148 podStartE2EDuration="4.313971133s" totalImagesPullingTime="2.310582985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:35:38.313144407 +0000 UTC m=+529.357530139" watchObservedRunningTime="2026-09-15 16:35:38.313971133 +0000 UTC m=+529.358356865"
Sep 15 16:36:09.306276 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:36:09.306246    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-z9sdp"
Sep 15 16:36:39.572190 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:36:39.572153    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:36:50.640512 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:36:50.640476    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:36:50.725078 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:36:50.725043    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:36:50.974414 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:36:50.974332    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:36:51.075155 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:36:51.075121    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:36:51.296203 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:36:51.296127    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:36:51.780148 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:36:51.780113    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:36:51.844495 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:36:51.844468    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:37:56.731017 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:37:56.730985    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:37:58.285843 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:37:58.285807    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:38:26.554228 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:26.554196    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:38:26.654035 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:26.654004    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:38:38.256499 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.256462    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:38:38.345512 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.345477    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:38:38.455697 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.455664    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:38:38.481511 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.481481    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"]
Sep 15 16:38:38.484503 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.484484    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:38.489202 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.489180    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dpw6w\""
Sep 15 16:38:38.489343 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.489326    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:38:38.489993 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.489972    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:38:38.490082 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.490013    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-19dbe-predictor-serving-cert\""
Sep 15 16:38:38.490082 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.490014    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-19dbe-kube-rbac-proxy-sar-config\""
Sep 15 16:38:38.495982 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.495959    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"]
Sep 15 16:38:38.518238 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.518167    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2pbx\" (UniqueName: \"kubernetes.io/projected/2ff81c6c-e922-4c67-bc1c-197a2f221750-kube-api-access-b2pbx\") pod \"success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp\" (UID: \"2ff81c6c-e922-4c67-bc1c-197a2f221750\") " pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:38.518238 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.518210    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff81c6c-e922-4c67-bc1c-197a2f221750-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp\" (UID: \"2ff81c6c-e922-4c67-bc1c-197a2f221750\") " pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:38.518434 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.518281    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff81c6c-e922-4c67-bc1c-197a2f221750-proxy-tls\") pod \"success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp\" (UID: \"2ff81c6c-e922-4c67-bc1c-197a2f221750\") " pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:38.566025 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.565992    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:38:38.618972 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.618941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff81c6c-e922-4c67-bc1c-197a2f221750-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp\" (UID: \"2ff81c6c-e922-4c67-bc1c-197a2f221750\") " pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:38.619162 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.619025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff81c6c-e922-4c67-bc1c-197a2f221750-proxy-tls\") pod \"success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp\" (UID: \"2ff81c6c-e922-4c67-bc1c-197a2f221750\") " pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:38.619162 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.619073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2pbx\" (UniqueName: \"kubernetes.io/projected/2ff81c6c-e922-4c67-bc1c-197a2f221750-kube-api-access-b2pbx\") pod \"success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp\" (UID: \"2ff81c6c-e922-4c67-bc1c-197a2f221750\") " pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:38.619514 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:38:38.619461    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-serving-cert: secret "success-200-isvc-19dbe-predictor-serving-cert" not found
Sep 15 16:38:38.619650 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:38:38.619565    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff81c6c-e922-4c67-bc1c-197a2f221750-proxy-tls podName:2ff81c6c-e922-4c67-bc1c-197a2f221750 nodeName:}" failed. No retries permitted until 2026-09-15 16:38:39.119545521 +0000 UTC m=+710.163931242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2ff81c6c-e922-4c67-bc1c-197a2f221750-proxy-tls") pod "success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" (UID: "2ff81c6c-e922-4c67-bc1c-197a2f221750") : secret "success-200-isvc-19dbe-predictor-serving-cert" not found
Sep 15 16:38:38.619809 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.619783    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff81c6c-e922-4c67-bc1c-197a2f221750-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp\" (UID: \"2ff81c6c-e922-4c67-bc1c-197a2f221750\") " pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:38.632788 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.632764    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2pbx\" (UniqueName: \"kubernetes.io/projected/2ff81c6c-e922-4c67-bc1c-197a2f221750-kube-api-access-b2pbx\") pod \"success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp\" (UID: \"2ff81c6c-e922-4c67-bc1c-197a2f221750\") " pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:38.666335 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.666302    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:38:38.677232 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.677210    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"]
Sep 15 16:38:38.680442 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.680427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:38.682618 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.682593    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-19dbe-predictor-serving-cert\""
Sep 15 16:38:38.682746 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.682701    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-19dbe-kube-rbac-proxy-sar-config\""
Sep 15 16:38:38.689124 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.689103    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"]
Sep 15 16:38:38.720045 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.720016    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/249d5b12-ea8c-4f9d-9742-4daa22a53276-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j\" (UID: \"249d5b12-ea8c-4f9d-9742-4daa22a53276\") " pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:38.720187 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.720079    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/249d5b12-ea8c-4f9d-9742-4daa22a53276-proxy-tls\") pod \"error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j\" (UID: \"249d5b12-ea8c-4f9d-9742-4daa22a53276\") " pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:38.720187 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.720114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c559\" (UniqueName: \"kubernetes.io/projected/249d5b12-ea8c-4f9d-9742-4daa22a53276-kube-api-access-8c559\") pod \"error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j\" (UID: \"249d5b12-ea8c-4f9d-9742-4daa22a53276\") " pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:38.765436 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.765405    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:38:38.820612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.820534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/249d5b12-ea8c-4f9d-9742-4daa22a53276-proxy-tls\") pod \"error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j\" (UID: \"249d5b12-ea8c-4f9d-9742-4daa22a53276\") " pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:38.820612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.820565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c559\" (UniqueName: \"kubernetes.io/projected/249d5b12-ea8c-4f9d-9742-4daa22a53276-kube-api-access-8c559\") pod \"error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j\" (UID: \"249d5b12-ea8c-4f9d-9742-4daa22a53276\") " pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:38.820826 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.820624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/249d5b12-ea8c-4f9d-9742-4daa22a53276-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j\" (UID: \"249d5b12-ea8c-4f9d-9742-4daa22a53276\") " pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:38.820826 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:38:38.820734    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-serving-cert: secret "error-404-isvc-19dbe-predictor-serving-cert" not found
Sep 15 16:38:38.820826 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:38:38.820816    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/249d5b12-ea8c-4f9d-9742-4daa22a53276-proxy-tls podName:249d5b12-ea8c-4f9d-9742-4daa22a53276 nodeName:}" failed. No retries permitted until 2026-09-15 16:38:39.320796026 +0000 UTC m=+710.365181741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/249d5b12-ea8c-4f9d-9742-4daa22a53276-proxy-tls") pod "error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" (UID: "249d5b12-ea8c-4f9d-9742-4daa22a53276") : secret "error-404-isvc-19dbe-predictor-serving-cert" not found
Sep 15 16:38:38.821312 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.821293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/249d5b12-ea8c-4f9d-9742-4daa22a53276-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j\" (UID: \"249d5b12-ea8c-4f9d-9742-4daa22a53276\") " pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:38.830734 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.830696    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c559\" (UniqueName: \"kubernetes.io/projected/249d5b12-ea8c-4f9d-9742-4daa22a53276-kube-api-access-8c559\") pod \"error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j\" (UID: \"249d5b12-ea8c-4f9d-9742-4daa22a53276\") " pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:38.865685 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:38.865650    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:38:39.010968 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:39.010940    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:38:39.111566 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:39.111469    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:38:39.124054 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:39.124032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff81c6c-e922-4c67-bc1c-197a2f221750-proxy-tls\") pod \"success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp\" (UID: \"2ff81c6c-e922-4c67-bc1c-197a2f221750\") " pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:39.126402 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:39.126380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff81c6c-e922-4c67-bc1c-197a2f221750-proxy-tls\") pod \"success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp\" (UID: \"2ff81c6c-e922-4c67-bc1c-197a2f221750\") " pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:39.212079 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:39.212041    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:38:39.326285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:39.326257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/249d5b12-ea8c-4f9d-9742-4daa22a53276-proxy-tls\") pod \"error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j\" (UID: \"249d5b12-ea8c-4f9d-9742-4daa22a53276\") " pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:39.328541 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:39.328520    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/249d5b12-ea8c-4f9d-9742-4daa22a53276-proxy-tls\") pod \"error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j\" (UID: \"249d5b12-ea8c-4f9d-9742-4daa22a53276\") " pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:39.394500 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:39.394465    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:39.592948 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:39.592910    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:39.664242 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:39.664166    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"]
Sep 15 16:38:39.666915 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:38:39.666888    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff81c6c_e922_4c67_bc1c_197a2f221750.slice/crio-3c03c49a72ea98e74c22235a1b20d8dd523813f33145cdd7be4d7df2a2d95e2c WatchSource:0}: Error finding container 3c03c49a72ea98e74c22235a1b20d8dd523813f33145cdd7be4d7df2a2d95e2c: Status 404 returned error can't find the container with id 3c03c49a72ea98e74c22235a1b20d8dd523813f33145cdd7be4d7df2a2d95e2c
Sep 15 16:38:39.884926 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:39.884889    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"]
Sep 15 16:38:39.887989 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:38:39.887961    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249d5b12_ea8c_4f9d_9742_4daa22a53276.slice/crio-7b4ff0db7e77ff7536a240223e3c74f8264f636273d335794999aea70a72c5e6 WatchSource:0}: Error finding container 7b4ff0db7e77ff7536a240223e3c74f8264f636273d335794999aea70a72c5e6: Status 404 returned error can't find the container with id 7b4ff0db7e77ff7536a240223e3c74f8264f636273d335794999aea70a72c5e6
Sep 15 16:38:39.897549 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:39.897521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" event={"ID":"249d5b12-ea8c-4f9d-9742-4daa22a53276","Type":"ContainerStarted","Data":"7b4ff0db7e77ff7536a240223e3c74f8264f636273d335794999aea70a72c5e6"}
Sep 15 16:38:39.898688 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:39.898669    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" event={"ID":"2ff81c6c-e922-4c67-bc1c-197a2f221750","Type":"ContainerStarted","Data":"3c03c49a72ea98e74c22235a1b20d8dd523813f33145cdd7be4d7df2a2d95e2c"}
Sep 15 16:38:55.985644 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:55.985549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" event={"ID":"249d5b12-ea8c-4f9d-9742-4daa22a53276","Type":"ContainerStarted","Data":"5d93039bf579005b2b1751a0d3f556a490669cfa6855e786f0ec8c7393ac9e76"}
Sep 15 16:38:55.986951 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:55.986926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" event={"ID":"2ff81c6c-e922-4c67-bc1c-197a2f221750","Type":"ContainerStarted","Data":"dad5c9c9906fc9e1f38ec4984607918a477bae246f940e7f3b27cb4e672925a5"}
Sep 15 16:38:59.001921 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:59.001884    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" event={"ID":"249d5b12-ea8c-4f9d-9742-4daa22a53276","Type":"ContainerStarted","Data":"c1d7790a38d3ab389d220cfb388b56f9d6abd04256e9ffedeb6ab734b38e2044"}
Sep 15 16:38:59.002350 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:59.002069    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:59.002350 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:59.002222    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:38:59.003545 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:59.003510    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 15 16:38:59.003874 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:59.003850    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" event={"ID":"2ff81c6c-e922-4c67-bc1c-197a2f221750","Type":"ContainerStarted","Data":"e650087ca3ef53ebd5a3ddfdbe8a3f96e54bd01bd6e8e979380672f3579eaec9"}
Sep 15 16:38:59.003969 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:59.003955    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:59.004009 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:59.003994    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:38:59.004995 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:59.004968    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 15 16:38:59.029316 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:38:59.029252    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" podStartSLOduration=2.62024803 podStartE2EDuration="21.029234717s" totalImagesPullingTime="18.408986687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:38:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:38:59.027586632 +0000 UTC m=+730.071972369" watchObservedRunningTime="2026-09-15 16:38:59.029234717 +0000 UTC m=+730.073620451"
Sep 15 16:39:00.007914 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:00.007868    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 15 16:39:00.008379 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:00.007962    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 15 16:39:05.012436 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:05.012403    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:39:05.012876 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:05.012694    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:39:05.012876 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:05.012832    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 15 16:39:05.013208 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:05.013187    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 15 16:39:05.031337 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:05.031285    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" podStartSLOduration=8.386469111 podStartE2EDuration="27.031272151s" totalImagesPullingTime="18.64480304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:38:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:38:59.057468726 +0000 UTC m=+730.101854457" watchObservedRunningTime="2026-09-15 16:39:05.031272151 +0000 UTC m=+736.075657883"
Sep 15 16:39:15.013216 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:15.013179    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 15 16:39:15.013811 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:15.013206    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 15 16:39:25.012892 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:25.012848    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 15 16:39:25.013292 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:25.013102    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 15 16:39:35.012968 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:35.012928    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 15 16:39:35.013398 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:35.013149    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 15 16:39:45.013703 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:45.013673    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:39:45.014087 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:45.013884    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:39:58.291145 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:58.291106    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:39:58.467871 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:39:58.467832    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:40:08.480079 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.480046    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:40:08.541071 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.541039    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"]
Sep 15 16:40:08.541313 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.541290    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kserve-container" containerID="cri-o://dad5c9c9906fc9e1f38ec4984607918a477bae246f940e7f3b27cb4e672925a5" gracePeriod=30
Sep 15 16:40:08.541393 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.541333    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kube-rbac-proxy" containerID="cri-o://e650087ca3ef53ebd5a3ddfdbe8a3f96e54bd01bd6e8e979380672f3579eaec9" gracePeriod=30
Sep 15 16:40:08.570978 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.570950    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:40:08.579909 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.579883    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"]
Sep 15 16:40:08.580126 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.580100    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kserve-container" containerID="cri-o://5d93039bf579005b2b1751a0d3f556a490669cfa6855e786f0ec8c7393ac9e76" gracePeriod=30
Sep 15 16:40:08.580255 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.580138    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kube-rbac-proxy" containerID="cri-o://c1d7790a38d3ab389d220cfb388b56f9d6abd04256e9ffedeb6ab734b38e2044" gracePeriod=30
Sep 15 16:40:08.651203 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.651171    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:40:08.665367 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.665332    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"]
Sep 15 16:40:08.668328 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.668308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:08.672115 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.672093    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cf726-predictor-serving-cert\""
Sep 15 16:40:08.672227 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.672131    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cf726-kube-rbac-proxy-sar-config\""
Sep 15 16:40:08.677570 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.677550    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"]
Sep 15 16:40:08.707318 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.707286    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c1cf7b8-db9b-4310-933f-f6d776baab6d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cf726-predictor-5d84565df6-lmbtw\" (UID: \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\") " pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:08.707473 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.707357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c1cf7b8-db9b-4310-933f-f6d776baab6d-proxy-tls\") pod \"success-200-isvc-cf726-predictor-5d84565df6-lmbtw\" (UID: \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\") " pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:08.707473 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.707464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwnqn\" (UniqueName: \"kubernetes.io/projected/5c1cf7b8-db9b-4310-933f-f6d776baab6d-kube-api-access-mwnqn\") pod \"success-200-isvc-cf726-predictor-5d84565df6-lmbtw\" (UID: \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\") " pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:08.750300 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.750216    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:40:08.755253 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.755228    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"]
Sep 15 16:40:08.758317 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.758301    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:08.760769 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.760740    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cf726-predictor-serving-cert\""
Sep 15 16:40:08.760769 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.760760    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cf726-kube-rbac-proxy-sar-config\""
Sep 15 16:40:08.767475 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.767452    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"]
Sep 15 16:40:08.808856 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.808828    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c1cf7b8-db9b-4310-933f-f6d776baab6d-proxy-tls\") pod \"success-200-isvc-cf726-predictor-5d84565df6-lmbtw\" (UID: \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\") " pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:08.808997 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.808865    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrl9v\" (UniqueName: \"kubernetes.io/projected/38261ae9-95a8-40a1-b2f8-9f5581655edb-kube-api-access-nrl9v\") pod \"error-404-isvc-cf726-predictor-68d66b987c-qrz2j\" (UID: \"38261ae9-95a8-40a1-b2f8-9f5581655edb\") " pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:08.808997 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.808887    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38261ae9-95a8-40a1-b2f8-9f5581655edb-proxy-tls\") pod \"error-404-isvc-cf726-predictor-68d66b987c-qrz2j\" (UID: \"38261ae9-95a8-40a1-b2f8-9f5581655edb\") " pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:08.808997 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.808922    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38261ae9-95a8-40a1-b2f8-9f5581655edb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cf726-predictor-68d66b987c-qrz2j\" (UID: \"38261ae9-95a8-40a1-b2f8-9f5581655edb\") " pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:08.808997 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.808970    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwnqn\" (UniqueName: \"kubernetes.io/projected/5c1cf7b8-db9b-4310-933f-f6d776baab6d-kube-api-access-mwnqn\") pod \"success-200-isvc-cf726-predictor-5d84565df6-lmbtw\" (UID: \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\") " pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:08.809216 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:40:08.809013    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-cf726-predictor-serving-cert: secret "success-200-isvc-cf726-predictor-serving-cert" not found
Sep 15 16:40:08.809216 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.809035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c1cf7b8-db9b-4310-933f-f6d776baab6d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cf726-predictor-5d84565df6-lmbtw\" (UID: \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\") " pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:08.809216 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:40:08.809095    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c1cf7b8-db9b-4310-933f-f6d776baab6d-proxy-tls podName:5c1cf7b8-db9b-4310-933f-f6d776baab6d nodeName:}" failed. No retries permitted until 2026-09-15 16:40:09.309071638 +0000 UTC m=+800.353457354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5c1cf7b8-db9b-4310-933f-f6d776baab6d-proxy-tls") pod "success-200-isvc-cf726-predictor-5d84565df6-lmbtw" (UID: "5c1cf7b8-db9b-4310-933f-f6d776baab6d") : secret "success-200-isvc-cf726-predictor-serving-cert" not found
Sep 15 16:40:08.809616 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.809596    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c1cf7b8-db9b-4310-933f-f6d776baab6d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cf726-predictor-5d84565df6-lmbtw\" (UID: \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\") " pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:08.819250 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.819221    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwnqn\" (UniqueName: \"kubernetes.io/projected/5c1cf7b8-db9b-4310-933f-f6d776baab6d-kube-api-access-mwnqn\") pod \"success-200-isvc-cf726-predictor-5d84565df6-lmbtw\" (UID: \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\") " pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:08.851556 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.851521    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:40:08.909632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.909596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38261ae9-95a8-40a1-b2f8-9f5581655edb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cf726-predictor-68d66b987c-qrz2j\" (UID: \"38261ae9-95a8-40a1-b2f8-9f5581655edb\") " pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:08.909786 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.909735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrl9v\" (UniqueName: \"kubernetes.io/projected/38261ae9-95a8-40a1-b2f8-9f5581655edb-kube-api-access-nrl9v\") pod \"error-404-isvc-cf726-predictor-68d66b987c-qrz2j\" (UID: \"38261ae9-95a8-40a1-b2f8-9f5581655edb\") " pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:08.909786 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.909767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38261ae9-95a8-40a1-b2f8-9f5581655edb-proxy-tls\") pod \"error-404-isvc-cf726-predictor-68d66b987c-qrz2j\" (UID: \"38261ae9-95a8-40a1-b2f8-9f5581655edb\") " pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:08.909903 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:40:08.909883    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-cf726-predictor-serving-cert: secret "error-404-isvc-cf726-predictor-serving-cert" not found
Sep 15 16:40:08.909946 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:40:08.909940    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38261ae9-95a8-40a1-b2f8-9f5581655edb-proxy-tls podName:38261ae9-95a8-40a1-b2f8-9f5581655edb nodeName:}" failed. No retries permitted until 2026-09-15 16:40:09.409921973 +0000 UTC m=+800.454307686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/38261ae9-95a8-40a1-b2f8-9f5581655edb-proxy-tls") pod "error-404-isvc-cf726-predictor-68d66b987c-qrz2j" (UID: "38261ae9-95a8-40a1-b2f8-9f5581655edb") : secret "error-404-isvc-cf726-predictor-serving-cert" not found
Sep 15 16:40:08.910360 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.910336    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38261ae9-95a8-40a1-b2f8-9f5581655edb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cf726-predictor-68d66b987c-qrz2j\" (UID: \"38261ae9-95a8-40a1-b2f8-9f5581655edb\") " pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:08.918874 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:08.918843    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrl9v\" (UniqueName: \"kubernetes.io/projected/38261ae9-95a8-40a1-b2f8-9f5581655edb-kube-api-access-nrl9v\") pod \"error-404-isvc-cf726-predictor-68d66b987c-qrz2j\" (UID: \"38261ae9-95a8-40a1-b2f8-9f5581655edb\") " pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:09.024688 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.024609    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:40:09.216534 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.216503    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:40:09.249480 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.249453    2565 generic.go:356] "Generic (PLEG): container finished" podID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerID="c1d7790a38d3ab389d220cfb388b56f9d6abd04256e9ffedeb6ab734b38e2044" exitCode=2
Sep 15 16:40:09.249644 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.249536    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" event={"ID":"249d5b12-ea8c-4f9d-9742-4daa22a53276","Type":"ContainerDied","Data":"c1d7790a38d3ab389d220cfb388b56f9d6abd04256e9ffedeb6ab734b38e2044"}
Sep 15 16:40:09.251394 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.251370    2565 generic.go:356] "Generic (PLEG): container finished" podID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerID="e650087ca3ef53ebd5a3ddfdbe8a3f96e54bd01bd6e8e979380672f3579eaec9" exitCode=2
Sep 15 16:40:09.251514 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.251443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" event={"ID":"2ff81c6c-e922-4c67-bc1c-197a2f221750","Type":"ContainerDied","Data":"e650087ca3ef53ebd5a3ddfdbe8a3f96e54bd01bd6e8e979380672f3579eaec9"}
Sep 15 16:40:09.314007 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.313932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c1cf7b8-db9b-4310-933f-f6d776baab6d-proxy-tls\") pod \"success-200-isvc-cf726-predictor-5d84565df6-lmbtw\" (UID: \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\") " pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:09.316267 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.316250    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c1cf7b8-db9b-4310-933f-f6d776baab6d-proxy-tls\") pod \"success-200-isvc-cf726-predictor-5d84565df6-lmbtw\" (UID: \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\") " pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:09.415049 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.415012    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38261ae9-95a8-40a1-b2f8-9f5581655edb-proxy-tls\") pod \"error-404-isvc-cf726-predictor-68d66b987c-qrz2j\" (UID: \"38261ae9-95a8-40a1-b2f8-9f5581655edb\") " pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:09.417364 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.417339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38261ae9-95a8-40a1-b2f8-9f5581655edb-proxy-tls\") pod \"error-404-isvc-cf726-predictor-68d66b987c-qrz2j\" (UID: \"38261ae9-95a8-40a1-b2f8-9f5581655edb\") " pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:09.580956 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.580874    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:09.671081 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.671046    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:09.850816 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.850669    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"]
Sep 15 16:40:09.853293 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:40:09.853263    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1cf7b8_db9b_4310_933f_f6d776baab6d.slice/crio-5c4c8d913135e7a23ba7141c25f0cdf9962541234d880b6791cda0cd3b67bc63 WatchSource:0}: Error finding container 5c4c8d913135e7a23ba7141c25f0cdf9962541234d880b6791cda0cd3b67bc63: Status 404 returned error can't find the container with id 5c4c8d913135e7a23ba7141c25f0cdf9962541234d880b6791cda0cd3b67bc63
Sep 15 16:40:09.939001 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:09.938950    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"]
Sep 15 16:40:09.942879 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:40:09.942847    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38261ae9_95a8_40a1_b2f8_9f5581655edb.slice/crio-abd67961a3ae8ef2445e361e5bfa8d5e1ef565b65ff660b7e198aa92fea13ab7 WatchSource:0}: Error finding container abd67961a3ae8ef2445e361e5bfa8d5e1ef565b65ff660b7e198aa92fea13ab7: Status 404 returned error can't find the container with id abd67961a3ae8ef2445e361e5bfa8d5e1ef565b65ff660b7e198aa92fea13ab7
Sep 15 16:40:10.008932 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:10.008903    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.15:8643/healthz\": dial tcp 10.134.0.15:8643: connect: connection refused"
Sep 15 16:40:10.009031 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:10.008905    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.16:8643/healthz\": dial tcp 10.134.0.16:8643: connect: connection refused"
Sep 15 16:40:10.257301 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:10.257259    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" event={"ID":"5c1cf7b8-db9b-4310-933f-f6d776baab6d","Type":"ContainerStarted","Data":"3c7a03e2a41ddb2bf28227e2167863aa2f11f832e2840809c48f6c8e2ca900b7"}
Sep 15 16:40:10.257482 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:10.257324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" event={"ID":"5c1cf7b8-db9b-4310-933f-f6d776baab6d","Type":"ContainerStarted","Data":"16a5588e12a8b80fb866b382c0296f4931de2377be18b1da9d23863a3b34888b"}
Sep 15 16:40:10.257482 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:10.257355    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" event={"ID":"5c1cf7b8-db9b-4310-933f-f6d776baab6d","Type":"ContainerStarted","Data":"5c4c8d913135e7a23ba7141c25f0cdf9962541234d880b6791cda0cd3b67bc63"}
Sep 15 16:40:10.257482 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:10.257388    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:10.259031 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:10.259003    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" event={"ID":"38261ae9-95a8-40a1-b2f8-9f5581655edb","Type":"ContainerStarted","Data":"253b8ceedc3d071f37141371f8876fee97bac1caf1b223b445bdfd4eff79265a"}
Sep 15 16:40:10.259123 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:10.259064    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:10.259123 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:10.259086    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" event={"ID":"38261ae9-95a8-40a1-b2f8-9f5581655edb","Type":"ContainerStarted","Data":"ba8aaf1bf4b04af2bcf94f117807a31350c35023e197fcf7825474a0ec30e754"}
Sep 15 16:40:10.259123 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:10.259107    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" event={"ID":"38261ae9-95a8-40a1-b2f8-9f5581655edb","Type":"ContainerStarted","Data":"abd67961a3ae8ef2445e361e5bfa8d5e1ef565b65ff660b7e198aa92fea13ab7"}
Sep 15 16:40:10.274260 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:10.274206    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" podStartSLOduration=2.274190105 podStartE2EDuration="2.274190105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:40:10.274155606 +0000 UTC m=+801.318541350" watchObservedRunningTime="2026-09-15 16:40:10.274190105 +0000 UTC m=+801.318575831"
Sep 15 16:40:11.262605 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:11.262575    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:11.263057 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:11.262631    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:11.263460 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:11.263434    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 15 16:40:11.263526 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:11.263500    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 15 16:40:12.244617 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.244591    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:40:12.247194 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.247175    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:40:12.261175 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.261124    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" podStartSLOduration=4.261107896 podStartE2EDuration="4.261107896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:40:10.292379003 +0000 UTC m=+801.336764738" watchObservedRunningTime="2026-09-15 16:40:12.261107896 +0000 UTC m=+803.305493627"
Sep 15 16:40:12.267550 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.267527    2565 generic.go:356] "Generic (PLEG): container finished" podID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerID="5d93039bf579005b2b1751a0d3f556a490669cfa6855e786f0ec8c7393ac9e76" exitCode=0
Sep 15 16:40:12.267887 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.267573    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"
Sep 15 16:40:12.267887 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.267608    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" event={"ID":"249d5b12-ea8c-4f9d-9742-4daa22a53276","Type":"ContainerDied","Data":"5d93039bf579005b2b1751a0d3f556a490669cfa6855e786f0ec8c7393ac9e76"}
Sep 15 16:40:12.267887 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.267688    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j" event={"ID":"249d5b12-ea8c-4f9d-9742-4daa22a53276","Type":"ContainerDied","Data":"7b4ff0db7e77ff7536a240223e3c74f8264f636273d335794999aea70a72c5e6"}
Sep 15 16:40:12.267887 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.267735    2565 scope.go:117] "RemoveContainer" containerID="c1d7790a38d3ab389d220cfb388b56f9d6abd04256e9ffedeb6ab734b38e2044"
Sep 15 16:40:12.269368 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.269338    2565 generic.go:356] "Generic (PLEG): container finished" podID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerID="dad5c9c9906fc9e1f38ec4984607918a477bae246f940e7f3b27cb4e672925a5" exitCode=0
Sep 15 16:40:12.269465 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.269395    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"
Sep 15 16:40:12.269465 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.269417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" event={"ID":"2ff81c6c-e922-4c67-bc1c-197a2f221750","Type":"ContainerDied","Data":"dad5c9c9906fc9e1f38ec4984607918a477bae246f940e7f3b27cb4e672925a5"}
Sep 15 16:40:12.269555 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.269475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp" event={"ID":"2ff81c6c-e922-4c67-bc1c-197a2f221750","Type":"ContainerDied","Data":"3c03c49a72ea98e74c22235a1b20d8dd523813f33145cdd7be4d7df2a2d95e2c"}
Sep 15 16:40:12.270000 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.269977    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 15 16:40:12.270103 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.270004    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 15 16:40:12.277115 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.277095    2565 scope.go:117] "RemoveContainer" containerID="5d93039bf579005b2b1751a0d3f556a490669cfa6855e786f0ec8c7393ac9e76"
Sep 15 16:40:12.286043 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.286021    2565 scope.go:117] "RemoveContainer" containerID="c1d7790a38d3ab389d220cfb388b56f9d6abd04256e9ffedeb6ab734b38e2044"
Sep 15 16:40:12.286331 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:40:12.286309    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d7790a38d3ab389d220cfb388b56f9d6abd04256e9ffedeb6ab734b38e2044\": container with ID starting with c1d7790a38d3ab389d220cfb388b56f9d6abd04256e9ffedeb6ab734b38e2044 not found: ID does not exist" containerID="c1d7790a38d3ab389d220cfb388b56f9d6abd04256e9ffedeb6ab734b38e2044"
Sep 15 16:40:12.286382 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.286343    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d7790a38d3ab389d220cfb388b56f9d6abd04256e9ffedeb6ab734b38e2044"} err="failed to get container status \"c1d7790a38d3ab389d220cfb388b56f9d6abd04256e9ffedeb6ab734b38e2044\": rpc error: code = NotFound desc = could not find container \"c1d7790a38d3ab389d220cfb388b56f9d6abd04256e9ffedeb6ab734b38e2044\": container with ID starting with c1d7790a38d3ab389d220cfb388b56f9d6abd04256e9ffedeb6ab734b38e2044 not found: ID does not exist"
Sep 15 16:40:12.286382 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.286372    2565 scope.go:117] "RemoveContainer" containerID="5d93039bf579005b2b1751a0d3f556a490669cfa6855e786f0ec8c7393ac9e76"
Sep 15 16:40:12.286610 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:40:12.286595    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d93039bf579005b2b1751a0d3f556a490669cfa6855e786f0ec8c7393ac9e76\": container with ID starting with 5d93039bf579005b2b1751a0d3f556a490669cfa6855e786f0ec8c7393ac9e76 not found: ID does not exist" containerID="5d93039bf579005b2b1751a0d3f556a490669cfa6855e786f0ec8c7393ac9e76"
Sep 15 16:40:12.286651 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.286613    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d93039bf579005b2b1751a0d3f556a490669cfa6855e786f0ec8c7393ac9e76"} err="failed to get container status \"5d93039bf579005b2b1751a0d3f556a490669cfa6855e786f0ec8c7393ac9e76\": rpc error: code = NotFound desc = could not find container \"5d93039bf579005b2b1751a0d3f556a490669cfa6855e786f0ec8c7393ac9e76\": container with ID starting with 5d93039bf579005b2b1751a0d3f556a490669cfa6855e786f0ec8c7393ac9e76 not found: ID does not exist"
Sep 15 16:40:12.286651 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.286626    2565 scope.go:117] "RemoveContainer" containerID="e650087ca3ef53ebd5a3ddfdbe8a3f96e54bd01bd6e8e979380672f3579eaec9"
Sep 15 16:40:12.293722 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.293689    2565 scope.go:117] "RemoveContainer" containerID="dad5c9c9906fc9e1f38ec4984607918a477bae246f940e7f3b27cb4e672925a5"
Sep 15 16:40:12.300576 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.300557    2565 scope.go:117] "RemoveContainer" containerID="e650087ca3ef53ebd5a3ddfdbe8a3f96e54bd01bd6e8e979380672f3579eaec9"
Sep 15 16:40:12.300955 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:40:12.300930    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e650087ca3ef53ebd5a3ddfdbe8a3f96e54bd01bd6e8e979380672f3579eaec9\": container with ID starting with e650087ca3ef53ebd5a3ddfdbe8a3f96e54bd01bd6e8e979380672f3579eaec9 not found: ID does not exist" containerID="e650087ca3ef53ebd5a3ddfdbe8a3f96e54bd01bd6e8e979380672f3579eaec9"
Sep 15 16:40:12.301007 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.300963    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e650087ca3ef53ebd5a3ddfdbe8a3f96e54bd01bd6e8e979380672f3579eaec9"} err="failed to get container status \"e650087ca3ef53ebd5a3ddfdbe8a3f96e54bd01bd6e8e979380672f3579eaec9\": rpc error: code = NotFound desc = could not find container \"e650087ca3ef53ebd5a3ddfdbe8a3f96e54bd01bd6e8e979380672f3579eaec9\": container with ID starting with e650087ca3ef53ebd5a3ddfdbe8a3f96e54bd01bd6e8e979380672f3579eaec9 not found: ID does not exist"
Sep 15 16:40:12.301007 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.300980    2565 scope.go:117] "RemoveContainer" containerID="dad5c9c9906fc9e1f38ec4984607918a477bae246f940e7f3b27cb4e672925a5"
Sep 15 16:40:12.301222 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:40:12.301205    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dad5c9c9906fc9e1f38ec4984607918a477bae246f940e7f3b27cb4e672925a5\": container with ID starting with dad5c9c9906fc9e1f38ec4984607918a477bae246f940e7f3b27cb4e672925a5 not found: ID does not exist" containerID="dad5c9c9906fc9e1f38ec4984607918a477bae246f940e7f3b27cb4e672925a5"
Sep 15 16:40:12.301267 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.301226    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dad5c9c9906fc9e1f38ec4984607918a477bae246f940e7f3b27cb4e672925a5"} err="failed to get container status \"dad5c9c9906fc9e1f38ec4984607918a477bae246f940e7f3b27cb4e672925a5\": rpc error: code = NotFound desc = could not find container \"dad5c9c9906fc9e1f38ec4984607918a477bae246f940e7f3b27cb4e672925a5\": container with ID starting with dad5c9c9906fc9e1f38ec4984607918a477bae246f940e7f3b27cb4e672925a5 not found: ID does not exist"
Sep 15 16:40:12.339170 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.339146    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff81c6c-e922-4c67-bc1c-197a2f221750-proxy-tls\") pod \"2ff81c6c-e922-4c67-bc1c-197a2f221750\" (UID: \"2ff81c6c-e922-4c67-bc1c-197a2f221750\") "
Sep 15 16:40:12.339285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.339206    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c559\" (UniqueName: \"kubernetes.io/projected/249d5b12-ea8c-4f9d-9742-4daa22a53276-kube-api-access-8c559\") pod \"249d5b12-ea8c-4f9d-9742-4daa22a53276\" (UID: \"249d5b12-ea8c-4f9d-9742-4daa22a53276\") "
Sep 15 16:40:12.339285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.339236    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/249d5b12-ea8c-4f9d-9742-4daa22a53276-kube-rbac-proxy-sar-config\") pod \"249d5b12-ea8c-4f9d-9742-4daa22a53276\" (UID: \"249d5b12-ea8c-4f9d-9742-4daa22a53276\") "
Sep 15 16:40:12.339285 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.339263    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff81c6c-e922-4c67-bc1c-197a2f221750-kube-rbac-proxy-sar-config\") pod \"2ff81c6c-e922-4c67-bc1c-197a2f221750\" (UID: \"2ff81c6c-e922-4c67-bc1c-197a2f221750\") "
Sep 15 16:40:12.339436 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.339295    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/249d5b12-ea8c-4f9d-9742-4daa22a53276-proxy-tls\") pod \"249d5b12-ea8c-4f9d-9742-4daa22a53276\" (UID: \"249d5b12-ea8c-4f9d-9742-4daa22a53276\") "
Sep 15 16:40:12.339436 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.339353    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2pbx\" (UniqueName: \"kubernetes.io/projected/2ff81c6c-e922-4c67-bc1c-197a2f221750-kube-api-access-b2pbx\") pod \"2ff81c6c-e922-4c67-bc1c-197a2f221750\" (UID: \"2ff81c6c-e922-4c67-bc1c-197a2f221750\") "
Sep 15 16:40:12.339740 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.339641    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/249d5b12-ea8c-4f9d-9742-4daa22a53276-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "249d5b12-ea8c-4f9d-9742-4daa22a53276" (UID: "249d5b12-ea8c-4f9d-9742-4daa22a53276"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:40:12.339740 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.339655    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff81c6c-e922-4c67-bc1c-197a2f221750-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ff81c6c-e922-4c67-bc1c-197a2f221750" (UID: "2ff81c6c-e922-4c67-bc1c-197a2f221750"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:40:12.341518 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.341488    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249d5b12-ea8c-4f9d-9742-4daa22a53276-kube-api-access-8c559" (OuterVolumeSpecName: "kube-api-access-8c559") pod "249d5b12-ea8c-4f9d-9742-4daa22a53276" (UID: "249d5b12-ea8c-4f9d-9742-4daa22a53276"). InnerVolumeSpecName "kube-api-access-8c559". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:40:12.341702 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.341676    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff81c6c-e922-4c67-bc1c-197a2f221750-kube-api-access-b2pbx" (OuterVolumeSpecName: "kube-api-access-b2pbx") pod "2ff81c6c-e922-4c67-bc1c-197a2f221750" (UID: "2ff81c6c-e922-4c67-bc1c-197a2f221750"). InnerVolumeSpecName "kube-api-access-b2pbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:40:12.341999 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.341983    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff81c6c-e922-4c67-bc1c-197a2f221750-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ff81c6c-e922-4c67-bc1c-197a2f221750" (UID: "2ff81c6c-e922-4c67-bc1c-197a2f221750"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:40:12.342045 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.341987    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/249d5b12-ea8c-4f9d-9742-4daa22a53276-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "249d5b12-ea8c-4f9d-9742-4daa22a53276" (UID: "249d5b12-ea8c-4f9d-9742-4daa22a53276"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:40:12.440573 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.440533    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2pbx\" (UniqueName: \"kubernetes.io/projected/2ff81c6c-e922-4c67-bc1c-197a2f221750-kube-api-access-b2pbx\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:40:12.440573 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.440575    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff81c6c-e922-4c67-bc1c-197a2f221750-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:40:12.440818 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.440593    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c559\" (UniqueName: \"kubernetes.io/projected/249d5b12-ea8c-4f9d-9742-4daa22a53276-kube-api-access-8c559\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:40:12.440818 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.440608    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/249d5b12-ea8c-4f9d-9742-4daa22a53276-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:40:12.440818 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.440624    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff81c6c-e922-4c67-bc1c-197a2f221750-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:40:12.440818 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.440638    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/249d5b12-ea8c-4f9d-9742-4daa22a53276-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:40:12.592546 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.592514    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"]
Sep 15 16:40:12.597424 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.597401    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-19dbe-predictor-96fddc7c8-lvhzp"]
Sep 15 16:40:12.608388 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.608360    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"]
Sep 15 16:40:12.614199 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:12.614177    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-19dbe-predictor-7bddfb8787-tsf9j"]
Sep 15 16:40:13.448942 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:13.448911    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" path="/var/lib/kubelet/pods/249d5b12-ea8c-4f9d-9742-4daa22a53276/volumes"
Sep 15 16:40:13.449320 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:13.449306    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" path="/var/lib/kubelet/pods/2ff81c6c-e922-4c67-bc1c-197a2f221750/volumes"
Sep 15 16:40:17.275054 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:17.275017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:17.275748 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:17.275694    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 15 16:40:17.276494 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:17.276470    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:17.276970 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:17.276934    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 15 16:40:27.276544 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:27.276507    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 15 16:40:27.277058 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:27.276952    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 15 16:40:37.276502 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:37.276461    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 15 16:40:37.276919 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:37.276894    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 15 16:40:47.276563 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:47.276527    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 15 16:40:47.276996 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:47.276970    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 15 16:40:48.242935 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:48.242897    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:40:49.173804 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:49.173771    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:40:57.275861 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:57.275831    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:40:57.277812 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:57.277795    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:40:58.585181 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.585147    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"]
Sep 15 16:40:58.585612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.585435    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kube-rbac-proxy"
Sep 15 16:40:58.585612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.585446    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kube-rbac-proxy"
Sep 15 16:40:58.585612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.585454    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kube-rbac-proxy"
Sep 15 16:40:58.585612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.585459    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kube-rbac-proxy"
Sep 15 16:40:58.585612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.585475    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kserve-container"
Sep 15 16:40:58.585612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.585480    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kserve-container"
Sep 15 16:40:58.585612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.585488    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kserve-container"
Sep 15 16:40:58.585612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.585493    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kserve-container"
Sep 15 16:40:58.585612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.585533    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kserve-container"
Sep 15 16:40:58.585612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.585541    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="249d5b12-ea8c-4f9d-9742-4daa22a53276" containerName="kube-rbac-proxy"
Sep 15 16:40:58.585612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.585548    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kserve-container"
Sep 15 16:40:58.585612 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.585552    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff81c6c-e922-4c67-bc1c-197a2f221750" containerName="kube-rbac-proxy"
Sep 15 16:40:58.589762 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.589745    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:40:58.592126 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.592108    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-daa19-predictor-serving-cert\""
Sep 15 16:40:58.592216 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.592135    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-daa19-kube-rbac-proxy-sar-config\""
Sep 15 16:40:58.598873 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.598848    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"]
Sep 15 16:40:58.679007 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.678968    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"]
Sep 15 16:40:58.682569 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.682550    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:40:58.683739 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.683700    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swhhj\" (UniqueName: \"kubernetes.io/projected/da838de0-317f-41ce-8768-d6ec3bd47a96-kube-api-access-swhhj\") pod \"success-200-isvc-daa19-predictor-78598b9488-nx46r\" (UID: \"da838de0-317f-41ce-8768-d6ec3bd47a96\") " pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:40:58.683837 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.683764    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da838de0-317f-41ce-8768-d6ec3bd47a96-proxy-tls\") pod \"success-200-isvc-daa19-predictor-78598b9488-nx46r\" (UID: \"da838de0-317f-41ce-8768-d6ec3bd47a96\") " pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:40:58.683837 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.683805    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da838de0-317f-41ce-8768-d6ec3bd47a96-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-daa19-predictor-78598b9488-nx46r\" (UID: \"da838de0-317f-41ce-8768-d6ec3bd47a96\") " pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:40:58.684943 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.684908    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-daa19-kube-rbac-proxy-sar-config\""
Sep 15 16:40:58.685045 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.684953    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-daa19-predictor-serving-cert\""
Sep 15 16:40:58.691425 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.691401    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"]
Sep 15 16:40:58.784878 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.784843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da838de0-317f-41ce-8768-d6ec3bd47a96-proxy-tls\") pod \"success-200-isvc-daa19-predictor-78598b9488-nx46r\" (UID: \"da838de0-317f-41ce-8768-d6ec3bd47a96\") " pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:40:58.784878 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.784891    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc4p2\" (UniqueName: \"kubernetes.io/projected/a09d582c-d45e-438b-b43c-14735137e8f1-kube-api-access-sc4p2\") pod \"error-404-isvc-daa19-predictor-64588c679c-fz4x9\" (UID: \"a09d582c-d45e-438b-b43c-14735137e8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:40:58.785128 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.784910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a09d582c-d45e-438b-b43c-14735137e8f1-proxy-tls\") pod \"error-404-isvc-daa19-predictor-64588c679c-fz4x9\" (UID: \"a09d582c-d45e-438b-b43c-14735137e8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:40:58.785128 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.784931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da838de0-317f-41ce-8768-d6ec3bd47a96-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-daa19-predictor-78598b9488-nx46r\" (UID: \"da838de0-317f-41ce-8768-d6ec3bd47a96\") " pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:40:58.785128 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.785017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swhhj\" (UniqueName: \"kubernetes.io/projected/da838de0-317f-41ce-8768-d6ec3bd47a96-kube-api-access-swhhj\") pod \"success-200-isvc-daa19-predictor-78598b9488-nx46r\" (UID: \"da838de0-317f-41ce-8768-d6ec3bd47a96\") " pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:40:58.785128 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.785053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a09d582c-d45e-438b-b43c-14735137e8f1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-daa19-predictor-64588c679c-fz4x9\" (UID: \"a09d582c-d45e-438b-b43c-14735137e8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:40:58.785563 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.785544    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da838de0-317f-41ce-8768-d6ec3bd47a96-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-daa19-predictor-78598b9488-nx46r\" (UID: \"da838de0-317f-41ce-8768-d6ec3bd47a96\") " pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:40:58.787324 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.787306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da838de0-317f-41ce-8768-d6ec3bd47a96-proxy-tls\") pod \"success-200-isvc-daa19-predictor-78598b9488-nx46r\" (UID: \"da838de0-317f-41ce-8768-d6ec3bd47a96\") " pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:40:58.792663 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.792636    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swhhj\" (UniqueName: \"kubernetes.io/projected/da838de0-317f-41ce-8768-d6ec3bd47a96-kube-api-access-swhhj\") pod \"success-200-isvc-daa19-predictor-78598b9488-nx46r\" (UID: \"da838de0-317f-41ce-8768-d6ec3bd47a96\") " pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:40:58.885882 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.885793    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a09d582c-d45e-438b-b43c-14735137e8f1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-daa19-predictor-64588c679c-fz4x9\" (UID: \"a09d582c-d45e-438b-b43c-14735137e8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:40:58.885882 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.885843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc4p2\" (UniqueName: \"kubernetes.io/projected/a09d582c-d45e-438b-b43c-14735137e8f1-kube-api-access-sc4p2\") pod \"error-404-isvc-daa19-predictor-64588c679c-fz4x9\" (UID: \"a09d582c-d45e-438b-b43c-14735137e8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:40:58.885882 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.885865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a09d582c-d45e-438b-b43c-14735137e8f1-proxy-tls\") pod \"error-404-isvc-daa19-predictor-64588c679c-fz4x9\" (UID: \"a09d582c-d45e-438b-b43c-14735137e8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:40:58.886116 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:40:58.885998    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-daa19-predictor-serving-cert: secret "error-404-isvc-daa19-predictor-serving-cert" not found
Sep 15 16:40:58.886116 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:40:58.886062    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a09d582c-d45e-438b-b43c-14735137e8f1-proxy-tls podName:a09d582c-d45e-438b-b43c-14735137e8f1 nodeName:}" failed. No retries permitted until 2026-09-15 16:40:59.386042929 +0000 UTC m=+850.430428639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a09d582c-d45e-438b-b43c-14735137e8f1-proxy-tls") pod "error-404-isvc-daa19-predictor-64588c679c-fz4x9" (UID: "a09d582c-d45e-438b-b43c-14735137e8f1") : secret "error-404-isvc-daa19-predictor-serving-cert" not found
Sep 15 16:40:58.886527 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.886506    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a09d582c-d45e-438b-b43c-14735137e8f1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-daa19-predictor-64588c679c-fz4x9\" (UID: \"a09d582c-d45e-438b-b43c-14735137e8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:40:58.894173 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.894152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc4p2\" (UniqueName: \"kubernetes.io/projected/a09d582c-d45e-438b-b43c-14735137e8f1-kube-api-access-sc4p2\") pod \"error-404-isvc-daa19-predictor-64588c679c-fz4x9\" (UID: \"a09d582c-d45e-438b-b43c-14735137e8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:40:58.900907 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:58.900883    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:40:59.173893 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:59.173833    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"]
Sep 15 16:40:59.176559 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:40:59.176533    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda838de0_317f_41ce_8768_d6ec3bd47a96.slice/crio-dd6bb4fb751189373e2a1b28518110f89a94b9ab0e1ed53b30c9ced4f38cea62 WatchSource:0}: Error finding container dd6bb4fb751189373e2a1b28518110f89a94b9ab0e1ed53b30c9ced4f38cea62: Status 404 returned error can't find the container with id dd6bb4fb751189373e2a1b28518110f89a94b9ab0e1ed53b30c9ced4f38cea62
Sep 15 16:40:59.389319 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:59.389284    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a09d582c-d45e-438b-b43c-14735137e8f1-proxy-tls\") pod \"error-404-isvc-daa19-predictor-64588c679c-fz4x9\" (UID: \"a09d582c-d45e-438b-b43c-14735137e8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:40:59.391681 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:59.391661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a09d582c-d45e-438b-b43c-14735137e8f1-proxy-tls\") pod \"error-404-isvc-daa19-predictor-64588c679c-fz4x9\" (UID: \"a09d582c-d45e-438b-b43c-14735137e8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:40:59.451281 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:59.451212    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:40:59.451281 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:59.451258    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" event={"ID":"da838de0-317f-41ce-8768-d6ec3bd47a96","Type":"ContainerStarted","Data":"1851b1e9c6caa8241759d33d31086bdffaa73a5b182880bb44f00f16f663e349"}
Sep 15 16:40:59.451281 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:59.451282    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" event={"ID":"da838de0-317f-41ce-8768-d6ec3bd47a96","Type":"ContainerStarted","Data":"76cfd594e2d4f24a5762fffd017e05ef01851fc77ae5f43f2ef3fbec439e442e"}
Sep 15 16:40:59.451474 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:59.451301    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" event={"ID":"da838de0-317f-41ce-8768-d6ec3bd47a96","Type":"ContainerStarted","Data":"dd6bb4fb751189373e2a1b28518110f89a94b9ab0e1ed53b30c9ced4f38cea62"}
Sep 15 16:40:59.467553 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:59.467507    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" podStartSLOduration=1.467491669 podStartE2EDuration="1.467491669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:40:59.465525477 +0000 UTC m=+850.509911213" watchObservedRunningTime="2026-09-15 16:40:59.467491669 +0000 UTC m=+850.511877398"
Sep 15 16:40:59.592449 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:59.592419    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:40:59.862007 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:59.861981    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"]
Sep 15 16:40:59.864560 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:40:59.864521    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09d582c_d45e_438b_b43c_14735137e8f1.slice/crio-5ca9c1f9b3b52ba75e25170f278a0df10469e3dabe56d2ea5962705080d68b75 WatchSource:0}: Error finding container 5ca9c1f9b3b52ba75e25170f278a0df10469e3dabe56d2ea5962705080d68b75: Status 404 returned error can't find the container with id 5ca9c1f9b3b52ba75e25170f278a0df10469e3dabe56d2ea5962705080d68b75
Sep 15 16:40:59.880264 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:59.880239    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:40:59.981230 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:40:59.981204    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:41:00.456089 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:00.456052    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" event={"ID":"a09d582c-d45e-438b-b43c-14735137e8f1","Type":"ContainerStarted","Data":"a4fccf354746614691ed8acd3bbc828f05342f1cb8f0604efbd2779995b2b2fd"}
Sep 15 16:41:00.456390 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:00.456374    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:41:00.456448 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:00.456419    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:41:00.456492 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:00.456446    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" event={"ID":"a09d582c-d45e-438b-b43c-14735137e8f1","Type":"ContainerStarted","Data":"44f4cbaa4d2bd6e988798d55ba6159422f7f07c3d4b871ee62b14c643d53ef0d"}
Sep 15 16:41:00.456492 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:00.456482    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:41:00.456563 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:00.456520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" event={"ID":"a09d582c-d45e-438b-b43c-14735137e8f1","Type":"ContainerStarted","Data":"5ca9c1f9b3b52ba75e25170f278a0df10469e3dabe56d2ea5962705080d68b75"}
Sep 15 16:41:00.457486 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:00.457458    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 15 16:41:00.457602 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:00.457486    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 15 16:41:00.474409 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:00.474371    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" podStartSLOduration=2.47435737 podStartE2EDuration="2.47435737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:41:00.473014589 +0000 UTC m=+851.517400322" watchObservedRunningTime="2026-09-15 16:41:00.47435737 +0000 UTC m=+851.518743102"
Sep 15 16:41:01.459546 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:01.459502    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 15 16:41:01.459943 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:01.459669    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 15 16:41:06.465276 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:06.465243    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:41:06.465729 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:06.465553    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:41:06.465847 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:06.465815    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 15 16:41:06.466177 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:06.466155    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 15 16:41:09.029476 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:09.029434    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:41:09.084176 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:09.084144    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:41:16.466652 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:16.466610    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 15 16:41:16.467057 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:16.466609    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 15 16:41:26.466657 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:26.466618    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 15 16:41:26.466657 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:26.466639    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 15 16:41:36.465912 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:36.465871    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 15 16:41:36.466429 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:36.466170    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 15 16:41:46.466865 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:46.466787    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:41:46.467227 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:46.466871    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:41:58.575648 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:58.575611    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:41:58.691721 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:41:58.691684    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:42:50.110789 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:42:50.110751    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:43:46.539265 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:43:46.539184    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:45:19.840339 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:45:19.840299    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:45:27.562064 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:45:27.562022    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:46:38.685521 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:46:38.685429    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:46:38.744198 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:46:38.744156    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:49:23.416679 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.416605    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:49:23.481469 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.481431    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"]
Sep 15 16:49:23.481762 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.481735    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kserve-container" containerID="cri-o://16a5588e12a8b80fb866b382c0296f4931de2377be18b1da9d23863a3b34888b" gracePeriod=30
Sep 15 16:49:23.481839 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.481798    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kube-rbac-proxy" containerID="cri-o://3c7a03e2a41ddb2bf28227e2167863aa2f11f832e2840809c48f6c8e2ca900b7" gracePeriod=30
Sep 15 16:49:23.649250 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.649214    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"]
Sep 15 16:49:23.652395 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.652379    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:23.654953 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.654927    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fcea7-predictor-serving-cert\""
Sep 15 16:49:23.655084 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.655043    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fcea7-kube-rbac-proxy-sar-config\""
Sep 15 16:49:23.665380 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.665354    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"]
Sep 15 16:49:23.669305 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.669241    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8\" (UID: \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\") " pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:23.669305 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.669298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt9vh\" (UniqueName: \"kubernetes.io/projected/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-kube-api-access-rt9vh\") pod \"success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8\" (UID: \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\") " pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:23.669468 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.669337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-proxy-tls\") pod \"success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8\" (UID: \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\") " pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:23.713894 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.713864    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"]
Sep 15 16:49:23.717081 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.717065    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:23.719479 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.719450    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fcea7-kube-rbac-proxy-sar-config\""
Sep 15 16:49:23.719607 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.719520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fcea7-predictor-serving-cert\""
Sep 15 16:49:23.726933 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.726908    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"]
Sep 15 16:49:23.754315 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.754272    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"]
Sep 15 16:49:23.754582 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.754554    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kserve-container" containerID="cri-o://ba8aaf1bf4b04af2bcf94f117807a31350c35023e197fcf7825474a0ec30e754" gracePeriod=30
Sep 15 16:49:23.754687 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.754588    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kube-rbac-proxy" containerID="cri-o://253b8ceedc3d071f37141371f8876fee97bac1caf1b223b445bdfd4eff79265a" gracePeriod=30
Sep 15 16:49:23.770536 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.770507    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt9vh\" (UniqueName: \"kubernetes.io/projected/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-kube-api-access-rt9vh\") pod \"success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8\" (UID: \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\") " pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:23.770638 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.770555    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-proxy-tls\") pod \"success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8\" (UID: \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\") " pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:23.770638 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.770602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e683e37-dc5f-459b-be23-2b621ad83e20-proxy-tls\") pod \"error-404-isvc-fcea7-predictor-597cb8b869-vsl4f\" (UID: \"5e683e37-dc5f-459b-be23-2b621ad83e20\") " pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:23.770808 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.770740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e683e37-dc5f-459b-be23-2b621ad83e20-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fcea7-predictor-597cb8b869-vsl4f\" (UID: \"5e683e37-dc5f-459b-be23-2b621ad83e20\") " pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:23.770808 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.770769    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gfgh\" (UniqueName: \"kubernetes.io/projected/5e683e37-dc5f-459b-be23-2b621ad83e20-kube-api-access-6gfgh\") pod \"error-404-isvc-fcea7-predictor-597cb8b869-vsl4f\" (UID: \"5e683e37-dc5f-459b-be23-2b621ad83e20\") " pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:23.770808 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.770801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8\" (UID: \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\") " pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:23.770964 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:49:23.770831    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-serving-cert: secret "success-200-isvc-fcea7-predictor-serving-cert" not found
Sep 15 16:49:23.770964 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:49:23.770931    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-proxy-tls podName:dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925 nodeName:}" failed. No retries permitted until 2026-09-15 16:49:24.270906412 +0000 UTC m=+1355.315292140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-proxy-tls") pod "success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" (UID: "dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925") : secret "success-200-isvc-fcea7-predictor-serving-cert" not found
Sep 15 16:49:23.771514 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.771489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8\" (UID: \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\") " pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:23.779839 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.779812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt9vh\" (UniqueName: \"kubernetes.io/projected/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-kube-api-access-rt9vh\") pod \"success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8\" (UID: \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\") " pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:23.871818 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.871785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e683e37-dc5f-459b-be23-2b621ad83e20-proxy-tls\") pod \"error-404-isvc-fcea7-predictor-597cb8b869-vsl4f\" (UID: \"5e683e37-dc5f-459b-be23-2b621ad83e20\") " pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:23.871818 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.871831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e683e37-dc5f-459b-be23-2b621ad83e20-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fcea7-predictor-597cb8b869-vsl4f\" (UID: \"5e683e37-dc5f-459b-be23-2b621ad83e20\") " pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:23.872066 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.871854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gfgh\" (UniqueName: \"kubernetes.io/projected/5e683e37-dc5f-459b-be23-2b621ad83e20-kube-api-access-6gfgh\") pod \"error-404-isvc-fcea7-predictor-597cb8b869-vsl4f\" (UID: \"5e683e37-dc5f-459b-be23-2b621ad83e20\") " pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:23.872066 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:49:23.871958    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-serving-cert: secret "error-404-isvc-fcea7-predictor-serving-cert" not found
Sep 15 16:49:23.872066 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:49:23.872031    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e683e37-dc5f-459b-be23-2b621ad83e20-proxy-tls podName:5e683e37-dc5f-459b-be23-2b621ad83e20 nodeName:}" failed. No retries permitted until 2026-09-15 16:49:24.372009248 +0000 UTC m=+1355.416394967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5e683e37-dc5f-459b-be23-2b621ad83e20-proxy-tls") pod "error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" (UID: "5e683e37-dc5f-459b-be23-2b621ad83e20") : secret "error-404-isvc-fcea7-predictor-serving-cert" not found
Sep 15 16:49:23.872534 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.872515    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e683e37-dc5f-459b-be23-2b621ad83e20-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fcea7-predictor-597cb8b869-vsl4f\" (UID: \"5e683e37-dc5f-459b-be23-2b621ad83e20\") " pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:23.882303 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:23.882274    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gfgh\" (UniqueName: \"kubernetes.io/projected/5e683e37-dc5f-459b-be23-2b621ad83e20-kube-api-access-6gfgh\") pod \"error-404-isvc-fcea7-predictor-597cb8b869-vsl4f\" (UID: \"5e683e37-dc5f-459b-be23-2b621ad83e20\") " pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:24.275524 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:24.275491    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-proxy-tls\") pod \"success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8\" (UID: \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\") " pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:24.278172 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:24.278143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-proxy-tls\") pod \"success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8\" (UID: \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\") " pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:24.281674 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:24.281648    2565 generic.go:356] "Generic (PLEG): container finished" podID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerID="3c7a03e2a41ddb2bf28227e2167863aa2f11f832e2840809c48f6c8e2ca900b7" exitCode=2
Sep 15 16:49:24.281803 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:24.281732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" event={"ID":"5c1cf7b8-db9b-4310-933f-f6d776baab6d","Type":"ContainerDied","Data":"3c7a03e2a41ddb2bf28227e2167863aa2f11f832e2840809c48f6c8e2ca900b7"}
Sep 15 16:49:24.283371 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:24.283347    2565 generic.go:356] "Generic (PLEG): container finished" podID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerID="253b8ceedc3d071f37141371f8876fee97bac1caf1b223b445bdfd4eff79265a" exitCode=2
Sep 15 16:49:24.283498 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:24.283413    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" event={"ID":"38261ae9-95a8-40a1-b2f8-9f5581655edb","Type":"ContainerDied","Data":"253b8ceedc3d071f37141371f8876fee97bac1caf1b223b445bdfd4eff79265a"}
Sep 15 16:49:24.375988 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:24.375948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e683e37-dc5f-459b-be23-2b621ad83e20-proxy-tls\") pod \"error-404-isvc-fcea7-predictor-597cb8b869-vsl4f\" (UID: \"5e683e37-dc5f-459b-be23-2b621ad83e20\") " pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:24.378338 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:24.378318    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e683e37-dc5f-459b-be23-2b621ad83e20-proxy-tls\") pod \"error-404-isvc-fcea7-predictor-597cb8b869-vsl4f\" (UID: \"5e683e37-dc5f-459b-be23-2b621ad83e20\") " pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:24.564477 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:24.564384    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:24.628265 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:24.628230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:24.837910 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:24.837887    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"]
Sep 15 16:49:24.839848 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:49:24.839809    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7b2af1_c7d9_4320_bbb7_4b5a5d6d0925.slice/crio-44abaa2203dc4ea58b7e287fcff9650d3d205e9a2a717e404149a1503fae1ba4 WatchSource:0}: Error finding container 44abaa2203dc4ea58b7e287fcff9650d3d205e9a2a717e404149a1503fae1ba4: Status 404 returned error can't find the container with id 44abaa2203dc4ea58b7e287fcff9650d3d205e9a2a717e404149a1503fae1ba4
Sep 15 16:49:24.900955 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:24.900921    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"]
Sep 15 16:49:24.903630 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:49:24.903609    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e683e37_dc5f_459b_be23_2b621ad83e20.slice/crio-b4d2090c74d0369bf28a776fb37693208e50196bb207ab54db7bf8e43fa0d72d WatchSource:0}: Error finding container b4d2090c74d0369bf28a776fb37693208e50196bb207ab54db7bf8e43fa0d72d: Status 404 returned error can't find the container with id b4d2090c74d0369bf28a776fb37693208e50196bb207ab54db7bf8e43fa0d72d
Sep 15 16:49:25.289904 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.289850    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" event={"ID":"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925","Type":"ContainerStarted","Data":"82a86615a51d4edddd417bbd72fba5bb393cc5d3fb1c6366bdce557363e79442"}
Sep 15 16:49:25.290096 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.289984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" event={"ID":"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925","Type":"ContainerStarted","Data":"f7c8ed9a436f71dbf3a64a50f9308fa0b2e8ba764931b0f0c7a516b39f9f7f50"}
Sep 15 16:49:25.290096 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.290029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" event={"ID":"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925","Type":"ContainerStarted","Data":"44abaa2203dc4ea58b7e287fcff9650d3d205e9a2a717e404149a1503fae1ba4"}
Sep 15 16:49:25.290096 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.290062    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:25.290096 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.290093    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:25.291131 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.291098    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 15 16:49:25.291832 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.291809    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" event={"ID":"5e683e37-dc5f-459b-be23-2b621ad83e20","Type":"ContainerStarted","Data":"915e651ce8751c9a1e15bcaf70d74a66649cbb9dc7bfccdbb450c20a2e84ffad"}
Sep 15 16:49:25.291935 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.291856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" event={"ID":"5e683e37-dc5f-459b-be23-2b621ad83e20","Type":"ContainerStarted","Data":"141729a823e41926287923fb090ee3e807fc1cdfc1ca488f29d4b71c72c80da8"}
Sep 15 16:49:25.291935 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.291877    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" event={"ID":"5e683e37-dc5f-459b-be23-2b621ad83e20","Type":"ContainerStarted","Data":"b4d2090c74d0369bf28a776fb37693208e50196bb207ab54db7bf8e43fa0d72d"}
Sep 15 16:49:25.291935 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.291902    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:25.291935 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.291922    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:25.293156 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.293132    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:49:25.306162 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.306117    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" podStartSLOduration=2.306103475 podStartE2EDuration="2.306103475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:49:25.305134885 +0000 UTC m=+1356.349520620" watchObservedRunningTime="2026-09-15 16:49:25.306103475 +0000 UTC m=+1356.350489208"
Sep 15 16:49:25.322150 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:25.322108    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" podStartSLOduration=2.322094568 podStartE2EDuration="2.322094568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:49:25.320523491 +0000 UTC m=+1356.364909235" watchObservedRunningTime="2026-09-15 16:49:25.322094568 +0000 UTC m=+1356.366480365"
Sep 15 16:49:26.295835 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.295789    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:49:26.296295 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.295918    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 15 16:49:26.453543 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.453501    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:49:26.722911 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.722890    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:49:26.792284 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.792251    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwnqn\" (UniqueName: \"kubernetes.io/projected/5c1cf7b8-db9b-4310-933f-f6d776baab6d-kube-api-access-mwnqn\") pod \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\" (UID: \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\") "
Sep 15 16:49:26.792444 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.792306    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c1cf7b8-db9b-4310-933f-f6d776baab6d-proxy-tls\") pod \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\" (UID: \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\") "
Sep 15 16:49:26.792444 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.792354    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c1cf7b8-db9b-4310-933f-f6d776baab6d-kube-rbac-proxy-sar-config\") pod \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\" (UID: \"5c1cf7b8-db9b-4310-933f-f6d776baab6d\") "
Sep 15 16:49:26.792807 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.792781    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1cf7b8-db9b-4310-933f-f6d776baab6d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c1cf7b8-db9b-4310-933f-f6d776baab6d" (UID: "5c1cf7b8-db9b-4310-933f-f6d776baab6d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:49:26.794426 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.794394    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1cf7b8-db9b-4310-933f-f6d776baab6d-kube-api-access-mwnqn" (OuterVolumeSpecName: "kube-api-access-mwnqn") pod "5c1cf7b8-db9b-4310-933f-f6d776baab6d" (UID: "5c1cf7b8-db9b-4310-933f-f6d776baab6d"). InnerVolumeSpecName "kube-api-access-mwnqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:49:26.794426 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.794413    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c1cf7b8-db9b-4310-933f-f6d776baab6d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c1cf7b8-db9b-4310-933f-f6d776baab6d" (UID: "5c1cf7b8-db9b-4310-933f-f6d776baab6d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:49:26.893050 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.893022    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwnqn\" (UniqueName: \"kubernetes.io/projected/5c1cf7b8-db9b-4310-933f-f6d776baab6d-kube-api-access-mwnqn\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:49:26.893050 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.893048    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c1cf7b8-db9b-4310-933f-f6d776baab6d-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:49:26.893050 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.893059    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c1cf7b8-db9b-4310-933f-f6d776baab6d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:49:26.982216 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:26.982179    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:49:27.094693 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.094652    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrl9v\" (UniqueName: \"kubernetes.io/projected/38261ae9-95a8-40a1-b2f8-9f5581655edb-kube-api-access-nrl9v\") pod \"38261ae9-95a8-40a1-b2f8-9f5581655edb\" (UID: \"38261ae9-95a8-40a1-b2f8-9f5581655edb\") "
Sep 15 16:49:27.094886 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.094739    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38261ae9-95a8-40a1-b2f8-9f5581655edb-proxy-tls\") pod \"38261ae9-95a8-40a1-b2f8-9f5581655edb\" (UID: \"38261ae9-95a8-40a1-b2f8-9f5581655edb\") "
Sep 15 16:49:27.094886 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.094772    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38261ae9-95a8-40a1-b2f8-9f5581655edb-kube-rbac-proxy-sar-config\") pod \"38261ae9-95a8-40a1-b2f8-9f5581655edb\" (UID: \"38261ae9-95a8-40a1-b2f8-9f5581655edb\") "
Sep 15 16:49:27.095181 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.095143    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38261ae9-95a8-40a1-b2f8-9f5581655edb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "38261ae9-95a8-40a1-b2f8-9f5581655edb" (UID: "38261ae9-95a8-40a1-b2f8-9f5581655edb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:49:27.096717 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.096688    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38261ae9-95a8-40a1-b2f8-9f5581655edb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "38261ae9-95a8-40a1-b2f8-9f5581655edb" (UID: "38261ae9-95a8-40a1-b2f8-9f5581655edb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:49:27.096796 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.096750    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38261ae9-95a8-40a1-b2f8-9f5581655edb-kube-api-access-nrl9v" (OuterVolumeSpecName: "kube-api-access-nrl9v") pod "38261ae9-95a8-40a1-b2f8-9f5581655edb" (UID: "38261ae9-95a8-40a1-b2f8-9f5581655edb"). InnerVolumeSpecName "kube-api-access-nrl9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:49:27.195388 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.195354    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38261ae9-95a8-40a1-b2f8-9f5581655edb-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:49:27.195388 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.195385    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38261ae9-95a8-40a1-b2f8-9f5581655edb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:49:27.195388 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.195395    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrl9v\" (UniqueName: \"kubernetes.io/projected/38261ae9-95a8-40a1-b2f8-9f5581655edb-kube-api-access-nrl9v\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:49:27.300374 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.300335    2565 generic.go:356] "Generic (PLEG): container finished" podID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerID="16a5588e12a8b80fb866b382c0296f4931de2377be18b1da9d23863a3b34888b" exitCode=0
Sep 15 16:49:27.300743 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.300386    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"
Sep 15 16:49:27.300743 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.300418    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" event={"ID":"5c1cf7b8-db9b-4310-933f-f6d776baab6d","Type":"ContainerDied","Data":"16a5588e12a8b80fb866b382c0296f4931de2377be18b1da9d23863a3b34888b"}
Sep 15 16:49:27.300743 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.300501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw" event={"ID":"5c1cf7b8-db9b-4310-933f-f6d776baab6d","Type":"ContainerDied","Data":"5c4c8d913135e7a23ba7141c25f0cdf9962541234d880b6791cda0cd3b67bc63"}
Sep 15 16:49:27.300743 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.300543    2565 scope.go:117] "RemoveContainer" containerID="3c7a03e2a41ddb2bf28227e2167863aa2f11f832e2840809c48f6c8e2ca900b7"
Sep 15 16:49:27.302164 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.302142    2565 generic.go:356] "Generic (PLEG): container finished" podID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerID="ba8aaf1bf4b04af2bcf94f117807a31350c35023e197fcf7825474a0ec30e754" exitCode=0
Sep 15 16:49:27.302254 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.302190    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"
Sep 15 16:49:27.302254 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.302192    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" event={"ID":"38261ae9-95a8-40a1-b2f8-9f5581655edb","Type":"ContainerDied","Data":"ba8aaf1bf4b04af2bcf94f117807a31350c35023e197fcf7825474a0ec30e754"}
Sep 15 16:49:27.302254 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.302238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j" event={"ID":"38261ae9-95a8-40a1-b2f8-9f5581655edb","Type":"ContainerDied","Data":"abd67961a3ae8ef2445e361e5bfa8d5e1ef565b65ff660b7e198aa92fea13ab7"}
Sep 15 16:49:27.311545 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.311514    2565 scope.go:117] "RemoveContainer" containerID="16a5588e12a8b80fb866b382c0296f4931de2377be18b1da9d23863a3b34888b"
Sep 15 16:49:27.319349 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.319334    2565 scope.go:117] "RemoveContainer" containerID="3c7a03e2a41ddb2bf28227e2167863aa2f11f832e2840809c48f6c8e2ca900b7"
Sep 15 16:49:27.319595 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:49:27.319580    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7a03e2a41ddb2bf28227e2167863aa2f11f832e2840809c48f6c8e2ca900b7\": container with ID starting with 3c7a03e2a41ddb2bf28227e2167863aa2f11f832e2840809c48f6c8e2ca900b7 not found: ID does not exist" containerID="3c7a03e2a41ddb2bf28227e2167863aa2f11f832e2840809c48f6c8e2ca900b7"
Sep 15 16:49:27.319633 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.319605    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7a03e2a41ddb2bf28227e2167863aa2f11f832e2840809c48f6c8e2ca900b7"} err="failed to get container status \"3c7a03e2a41ddb2bf28227e2167863aa2f11f832e2840809c48f6c8e2ca900b7\": rpc error: code = NotFound desc = could not find container \"3c7a03e2a41ddb2bf28227e2167863aa2f11f832e2840809c48f6c8e2ca900b7\": container with ID starting with 3c7a03e2a41ddb2bf28227e2167863aa2f11f832e2840809c48f6c8e2ca900b7 not found: ID does not exist"
Sep 15 16:49:27.319633 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.319620    2565 scope.go:117] "RemoveContainer" containerID="16a5588e12a8b80fb866b382c0296f4931de2377be18b1da9d23863a3b34888b"
Sep 15 16:49:27.319859 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:49:27.319841    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a5588e12a8b80fb866b382c0296f4931de2377be18b1da9d23863a3b34888b\": container with ID starting with 16a5588e12a8b80fb866b382c0296f4931de2377be18b1da9d23863a3b34888b not found: ID does not exist" containerID="16a5588e12a8b80fb866b382c0296f4931de2377be18b1da9d23863a3b34888b"
Sep 15 16:49:27.319928 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.319867    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a5588e12a8b80fb866b382c0296f4931de2377be18b1da9d23863a3b34888b"} err="failed to get container status \"16a5588e12a8b80fb866b382c0296f4931de2377be18b1da9d23863a3b34888b\": rpc error: code = NotFound desc = could not find container \"16a5588e12a8b80fb866b382c0296f4931de2377be18b1da9d23863a3b34888b\": container with ID starting with 16a5588e12a8b80fb866b382c0296f4931de2377be18b1da9d23863a3b34888b not found: ID does not exist"
Sep 15 16:49:27.319928 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.319889    2565 scope.go:117] "RemoveContainer" containerID="253b8ceedc3d071f37141371f8876fee97bac1caf1b223b445bdfd4eff79265a"
Sep 15 16:49:27.327263 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.327244    2565 scope.go:117] "RemoveContainer" containerID="ba8aaf1bf4b04af2bcf94f117807a31350c35023e197fcf7825474a0ec30e754"
Sep 15 16:49:27.327496 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.327476    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"]
Sep 15 16:49:27.331529 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.331511    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cf726-predictor-5d84565df6-lmbtw"]
Sep 15 16:49:27.334080 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.334058    2565 scope.go:117] "RemoveContainer" containerID="253b8ceedc3d071f37141371f8876fee97bac1caf1b223b445bdfd4eff79265a"
Sep 15 16:49:27.334358 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:49:27.334338    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253b8ceedc3d071f37141371f8876fee97bac1caf1b223b445bdfd4eff79265a\": container with ID starting with 253b8ceedc3d071f37141371f8876fee97bac1caf1b223b445bdfd4eff79265a not found: ID does not exist" containerID="253b8ceedc3d071f37141371f8876fee97bac1caf1b223b445bdfd4eff79265a"
Sep 15 16:49:27.334432 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.334364    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253b8ceedc3d071f37141371f8876fee97bac1caf1b223b445bdfd4eff79265a"} err="failed to get container status \"253b8ceedc3d071f37141371f8876fee97bac1caf1b223b445bdfd4eff79265a\": rpc error: code = NotFound desc = could not find container \"253b8ceedc3d071f37141371f8876fee97bac1caf1b223b445bdfd4eff79265a\": container with ID starting with 253b8ceedc3d071f37141371f8876fee97bac1caf1b223b445bdfd4eff79265a not found: ID does not exist"
Sep 15 16:49:27.334432 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.334383    2565 scope.go:117] "RemoveContainer" containerID="ba8aaf1bf4b04af2bcf94f117807a31350c35023e197fcf7825474a0ec30e754"
Sep 15 16:49:27.334650 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:49:27.334634    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8aaf1bf4b04af2bcf94f117807a31350c35023e197fcf7825474a0ec30e754\": container with ID starting with ba8aaf1bf4b04af2bcf94f117807a31350c35023e197fcf7825474a0ec30e754 not found: ID does not exist" containerID="ba8aaf1bf4b04af2bcf94f117807a31350c35023e197fcf7825474a0ec30e754"
Sep 15 16:49:27.334764 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.334656    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8aaf1bf4b04af2bcf94f117807a31350c35023e197fcf7825474a0ec30e754"} err="failed to get container status \"ba8aaf1bf4b04af2bcf94f117807a31350c35023e197fcf7825474a0ec30e754\": rpc error: code = NotFound desc = could not find container \"ba8aaf1bf4b04af2bcf94f117807a31350c35023e197fcf7825474a0ec30e754\": container with ID starting with ba8aaf1bf4b04af2bcf94f117807a31350c35023e197fcf7825474a0ec30e754 not found: ID does not exist"
Sep 15 16:49:27.340169 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.340149    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"]
Sep 15 16:49:27.342035 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.342016    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cf726-predictor-68d66b987c-qrz2j"]
Sep 15 16:49:27.449663 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.449573    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" path="/var/lib/kubelet/pods/38261ae9-95a8-40a1-b2f8-9f5581655edb/volumes"
Sep 15 16:49:27.450114 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:27.450093    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" path="/var/lib/kubelet/pods/5c1cf7b8-db9b-4310-933f-f6d776baab6d/volumes"
Sep 15 16:49:31.301137 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:31.301109    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:49:31.301484 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:31.301191    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:49:31.301673 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:31.301646    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:49:31.301987 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:31.301959    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 15 16:49:41.301901 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:41.301855    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:49:41.302290 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:41.301994    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 15 16:49:51.301819 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:51.301779    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:49:51.302205 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:49:51.301913    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 15 16:50:01.302226 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:01.302185    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:50:01.302796 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:01.302187    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 15 16:50:11.302364 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:11.302329    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:50:11.302811 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:11.302465    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:50:13.427561 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.427527    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"]
Sep 15 16:50:13.427975 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.427826    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kserve-container" containerID="cri-o://76cfd594e2d4f24a5762fffd017e05ef01851fc77ae5f43f2ef3fbec439e442e" gracePeriod=30
Sep 15 16:50:13.427975 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.427903    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kube-rbac-proxy" containerID="cri-o://1851b1e9c6caa8241759d33d31086bdffaa73a5b182880bb44f00f16f663e349" gracePeriod=30
Sep 15 16:50:13.514272 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.514235    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"]
Sep 15 16:50:13.514533 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.514522    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kserve-container"
Sep 15 16:50:13.514574 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.514536    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kserve-container"
Sep 15 16:50:13.514574 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.514547    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kserve-container"
Sep 15 16:50:13.514574 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.514552    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kserve-container"
Sep 15 16:50:13.514574 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.514559    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kube-rbac-proxy"
Sep 15 16:50:13.514574 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.514564    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kube-rbac-proxy"
Sep 15 16:50:13.514574 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.514572    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kube-rbac-proxy"
Sep 15 16:50:13.514574 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.514577    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kube-rbac-proxy"
Sep 15 16:50:13.514800 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.514624    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kube-rbac-proxy"
Sep 15 16:50:13.514800 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.514632    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kserve-container"
Sep 15 16:50:13.514800 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.514640    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c1cf7b8-db9b-4310-933f-f6d776baab6d" containerName="kserve-container"
Sep 15 16:50:13.514800 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.514646    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="38261ae9-95a8-40a1-b2f8-9f5581655edb" containerName="kube-rbac-proxy"
Sep 15 16:50:13.518850 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.518833    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:13.521357 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.521331    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1d1fb-predictor-serving-cert\""
Sep 15 16:50:13.521475 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.521332    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1d1fb-kube-rbac-proxy-sar-config\""
Sep 15 16:50:13.527549 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.527523    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"]
Sep 15 16:50:13.549112 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.549083    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"]
Sep 15 16:50:13.549343 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.549313    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kserve-container" containerID="cri-o://44f4cbaa4d2bd6e988798d55ba6159422f7f07c3d4b871ee62b14c643d53ef0d" gracePeriod=30
Sep 15 16:50:13.549468 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.549376    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kube-rbac-proxy" containerID="cri-o://a4fccf354746614691ed8acd3bbc828f05342f1cb8f0604efbd2779995b2b2fd" gracePeriod=30
Sep 15 16:50:13.640747 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.640700    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"]
Sep 15 16:50:13.644325 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.644308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:13.646696 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.646675    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1d1fb-kube-rbac-proxy-sar-config\""
Sep 15 16:50:13.646696 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.646686    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1d1fb-predictor-serving-cert\""
Sep 15 16:50:13.649958 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.649937    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90c20cef-c135-4127-9b1b-fbf02e97ad25-proxy-tls\") pod \"success-200-isvc-1d1fb-predictor-7c46f69ccb-95727\" (UID: \"90c20cef-c135-4127-9b1b-fbf02e97ad25\") " pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:13.650072 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.649983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxp9k\" (UniqueName: \"kubernetes.io/projected/90c20cef-c135-4127-9b1b-fbf02e97ad25-kube-api-access-dxp9k\") pod \"success-200-isvc-1d1fb-predictor-7c46f69ccb-95727\" (UID: \"90c20cef-c135-4127-9b1b-fbf02e97ad25\") " pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:13.650072 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.650012    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90c20cef-c135-4127-9b1b-fbf02e97ad25-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1d1fb-predictor-7c46f69ccb-95727\" (UID: \"90c20cef-c135-4127-9b1b-fbf02e97ad25\") " pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:13.652604 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.652576    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"]
Sep 15 16:50:13.751520 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.751431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm\" (UID: \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\") " pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:13.751520 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.751473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9lsn\" (UniqueName: \"kubernetes.io/projected/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-kube-api-access-f9lsn\") pod \"error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm\" (UID: \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\") " pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:13.751766 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.751536    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90c20cef-c135-4127-9b1b-fbf02e97ad25-proxy-tls\") pod \"success-200-isvc-1d1fb-predictor-7c46f69ccb-95727\" (UID: \"90c20cef-c135-4127-9b1b-fbf02e97ad25\") " pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:13.751766 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.751575    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-proxy-tls\") pod \"error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm\" (UID: \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\") " pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:13.751766 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:13.751597    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-serving-cert: secret "success-200-isvc-1d1fb-predictor-serving-cert" not found
Sep 15 16:50:13.751766 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.751613    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxp9k\" (UniqueName: \"kubernetes.io/projected/90c20cef-c135-4127-9b1b-fbf02e97ad25-kube-api-access-dxp9k\") pod \"success-200-isvc-1d1fb-predictor-7c46f69ccb-95727\" (UID: \"90c20cef-c135-4127-9b1b-fbf02e97ad25\") " pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:13.751766 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:13.751645    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c20cef-c135-4127-9b1b-fbf02e97ad25-proxy-tls podName:90c20cef-c135-4127-9b1b-fbf02e97ad25 nodeName:}" failed. No retries permitted until 2026-09-15 16:50:14.251629654 +0000 UTC m=+1405.296015364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/90c20cef-c135-4127-9b1b-fbf02e97ad25-proxy-tls") pod "success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" (UID: "90c20cef-c135-4127-9b1b-fbf02e97ad25") : secret "success-200-isvc-1d1fb-predictor-serving-cert" not found
Sep 15 16:50:13.751766 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.751664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90c20cef-c135-4127-9b1b-fbf02e97ad25-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1d1fb-predictor-7c46f69ccb-95727\" (UID: \"90c20cef-c135-4127-9b1b-fbf02e97ad25\") " pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:13.752312 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.752292    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90c20cef-c135-4127-9b1b-fbf02e97ad25-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1d1fb-predictor-7c46f69ccb-95727\" (UID: \"90c20cef-c135-4127-9b1b-fbf02e97ad25\") " pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:13.760403 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.760384    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxp9k\" (UniqueName: \"kubernetes.io/projected/90c20cef-c135-4127-9b1b-fbf02e97ad25-kube-api-access-dxp9k\") pod \"success-200-isvc-1d1fb-predictor-7c46f69ccb-95727\" (UID: \"90c20cef-c135-4127-9b1b-fbf02e97ad25\") " pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:13.774761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.774736    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:13.832660 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.832631    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:13.852077 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.852053    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-proxy-tls\") pod \"error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm\" (UID: \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\") " pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:13.852221 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.852123    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm\" (UID: \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\") " pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:13.852221 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.852158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9lsn\" (UniqueName: \"kubernetes.io/projected/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-kube-api-access-f9lsn\") pod \"error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm\" (UID: \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\") " pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:13.852332 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:13.852218    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-serving-cert: secret "error-404-isvc-1d1fb-predictor-serving-cert" not found
Sep 15 16:50:13.852332 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:13.852293    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-proxy-tls podName:e6e325ea-ffa7-42b0-ab3a-c5135cb72073 nodeName:}" failed. No retries permitted until 2026-09-15 16:50:14.35227085 +0000 UTC m=+1405.396656577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-proxy-tls") pod "error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" (UID: "e6e325ea-ffa7-42b0-ab3a-c5135cb72073") : secret "error-404-isvc-1d1fb-predictor-serving-cert" not found
Sep 15 16:50:13.852910 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.852888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm\" (UID: \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\") " pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:13.861676 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.861652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9lsn\" (UniqueName: \"kubernetes.io/projected/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-kube-api-access-f9lsn\") pod \"error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm\" (UID: \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\") " pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:13.914584 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:13.914554    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:14.015690 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:14.015609    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:14.255514 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:14.255472    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90c20cef-c135-4127-9b1b-fbf02e97ad25-proxy-tls\") pod \"success-200-isvc-1d1fb-predictor-7c46f69ccb-95727\" (UID: \"90c20cef-c135-4127-9b1b-fbf02e97ad25\") " pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:14.257937 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:14.257909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90c20cef-c135-4127-9b1b-fbf02e97ad25-proxy-tls\") pod \"success-200-isvc-1d1fb-predictor-7c46f69ccb-95727\" (UID: \"90c20cef-c135-4127-9b1b-fbf02e97ad25\") " pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:14.356963 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:14.356878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-proxy-tls\") pod \"error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm\" (UID: \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\") " pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:14.359177 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:14.359157    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-proxy-tls\") pod \"error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm\" (UID: \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\") " pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:14.430835 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:14.430795    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:14.481118 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:14.480950    2565 generic.go:356] "Generic (PLEG): container finished" podID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerID="1851b1e9c6caa8241759d33d31086bdffaa73a5b182880bb44f00f16f663e349" exitCode=2
Sep 15 16:50:14.481118 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:14.481040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" event={"ID":"da838de0-317f-41ce-8768-d6ec3bd47a96","Type":"ContainerDied","Data":"1851b1e9c6caa8241759d33d31086bdffaa73a5b182880bb44f00f16f663e349"}
Sep 15 16:50:14.484021 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:14.483943    2565 generic.go:356] "Generic (PLEG): container finished" podID="a09d582c-d45e-438b-b43c-14735137e8f1" containerID="a4fccf354746614691ed8acd3bbc828f05342f1cb8f0604efbd2779995b2b2fd" exitCode=2
Sep 15 16:50:14.484133 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:14.484026    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" event={"ID":"a09d582c-d45e-438b-b43c-14735137e8f1","Type":"ContainerDied","Data":"a4fccf354746614691ed8acd3bbc828f05342f1cb8f0604efbd2779995b2b2fd"}
Sep 15 16:50:14.555586 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:14.555547    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:14.703848 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:14.703821    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"]
Sep 15 16:50:14.706003 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:50:14.705977    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c20cef_c135_4127_9b1b_fbf02e97ad25.slice/crio-d0051314872e1811d01db7f2b01bf1bfe3f9f1e2221a0a1f422c091e2124ae7d WatchSource:0}: Error finding container d0051314872e1811d01db7f2b01bf1bfe3f9f1e2221a0a1f422c091e2124ae7d: Status 404 returned error can't find the container with id d0051314872e1811d01db7f2b01bf1bfe3f9f1e2221a0a1f422c091e2124ae7d
Sep 15 16:50:14.829027 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:14.829001    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"]
Sep 15 16:50:14.831460 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:50:14.831429    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e325ea_ffa7_42b0_ab3a_c5135cb72073.slice/crio-1a09eebf42af86bde9f12a5b0455a88a11ded7c3405b09c63baf40c277268404 WatchSource:0}: Error finding container 1a09eebf42af86bde9f12a5b0455a88a11ded7c3405b09c63baf40c277268404: Status 404 returned error can't find the container with id 1a09eebf42af86bde9f12a5b0455a88a11ded7c3405b09c63baf40c277268404
Sep 15 16:50:15.489152 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.489113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" event={"ID":"90c20cef-c135-4127-9b1b-fbf02e97ad25","Type":"ContainerStarted","Data":"6029a319f09b1a8b720bb55dd73091f332bf1211abbf554c64dc2841f486c495"}
Sep 15 16:50:15.489593 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.489181    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:15.489593 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.489214    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:15.489593 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.489239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" event={"ID":"90c20cef-c135-4127-9b1b-fbf02e97ad25","Type":"ContainerStarted","Data":"13e1c2c136a9211af88c10e5a5526b22ad2100fcc48cf08c195027264ef36c77"}
Sep 15 16:50:15.489593 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.489267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" event={"ID":"90c20cef-c135-4127-9b1b-fbf02e97ad25","Type":"ContainerStarted","Data":"d0051314872e1811d01db7f2b01bf1bfe3f9f1e2221a0a1f422c091e2124ae7d"}
Sep 15 16:50:15.490913 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.490885    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:50:15.491058 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.490947    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" event={"ID":"e6e325ea-ffa7-42b0-ab3a-c5135cb72073","Type":"ContainerStarted","Data":"49329ebe531f8fdd64e0331fcbbf7bd76ce52b2203394ad9ba6e46ab191f455d"}
Sep 15 16:50:15.491058 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.491001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" event={"ID":"e6e325ea-ffa7-42b0-ab3a-c5135cb72073","Type":"ContainerStarted","Data":"4c1ff538b69ea9ab0dc46c2c8f76233c81e5e4fb813ed2bea26e4128564181c6"}
Sep 15 16:50:15.491058 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.491030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" event={"ID":"e6e325ea-ffa7-42b0-ab3a-c5135cb72073","Type":"ContainerStarted","Data":"1a09eebf42af86bde9f12a5b0455a88a11ded7c3405b09c63baf40c277268404"}
Sep 15 16:50:15.491221 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.491060    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:15.491221 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.491092    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:15.491905 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.491884    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:50:15.508360 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.508317    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" podStartSLOduration=2.508302922 podStartE2EDuration="2.508302922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:50:15.506182373 +0000 UTC m=+1406.550568118" watchObservedRunningTime="2026-09-15 16:50:15.508302922 +0000 UTC m=+1406.552688725"
Sep 15 16:50:15.523019 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:15.522980    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" podStartSLOduration=2.5229671099999997 podStartE2EDuration="2.52296711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:50:15.522390915 +0000 UTC m=+1406.566776648" watchObservedRunningTime="2026-09-15 16:50:15.52296711 +0000 UTC m=+1406.567352843"
Sep 15 16:50:16.460140 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:16.460091    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Sep 15 16:50:16.460307 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:16.460091    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Sep 15 16:50:16.466222 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:16.466197    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 15 16:50:16.466282 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:16.466205    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 15 16:50:16.495290 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:16.495258    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:50:16.495626 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:16.495325    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:50:16.894905 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:16.894882    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:50:16.975810 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:16.975772    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a09d582c-d45e-438b-b43c-14735137e8f1-kube-rbac-proxy-sar-config\") pod \"a09d582c-d45e-438b-b43c-14735137e8f1\" (UID: \"a09d582c-d45e-438b-b43c-14735137e8f1\") "
Sep 15 16:50:16.975982 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:16.975838    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc4p2\" (UniqueName: \"kubernetes.io/projected/a09d582c-d45e-438b-b43c-14735137e8f1-kube-api-access-sc4p2\") pod \"a09d582c-d45e-438b-b43c-14735137e8f1\" (UID: \"a09d582c-d45e-438b-b43c-14735137e8f1\") "
Sep 15 16:50:16.975982 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:16.975867    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a09d582c-d45e-438b-b43c-14735137e8f1-proxy-tls\") pod \"a09d582c-d45e-438b-b43c-14735137e8f1\" (UID: \"a09d582c-d45e-438b-b43c-14735137e8f1\") "
Sep 15 16:50:16.976176 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:16.976145    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09d582c-d45e-438b-b43c-14735137e8f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a09d582c-d45e-438b-b43c-14735137e8f1" (UID: "a09d582c-d45e-438b-b43c-14735137e8f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:50:16.977994 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:16.977973    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09d582c-d45e-438b-b43c-14735137e8f1-kube-api-access-sc4p2" (OuterVolumeSpecName: "kube-api-access-sc4p2") pod "a09d582c-d45e-438b-b43c-14735137e8f1" (UID: "a09d582c-d45e-438b-b43c-14735137e8f1"). InnerVolumeSpecName "kube-api-access-sc4p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:50:16.978071 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:16.978038    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a09d582c-d45e-438b-b43c-14735137e8f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a09d582c-d45e-438b-b43c-14735137e8f1" (UID: "a09d582c-d45e-438b-b43c-14735137e8f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:50:17.076697 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.076663    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc4p2\" (UniqueName: \"kubernetes.io/projected/a09d582c-d45e-438b-b43c-14735137e8f1-kube-api-access-sc4p2\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:50:17.076697 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.076691    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a09d582c-d45e-438b-b43c-14735137e8f1-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:50:17.076697 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.076703    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a09d582c-d45e-438b-b43c-14735137e8f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:50:17.265057 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.265032    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:50:17.379672 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.379580    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da838de0-317f-41ce-8768-d6ec3bd47a96-proxy-tls\") pod \"da838de0-317f-41ce-8768-d6ec3bd47a96\" (UID: \"da838de0-317f-41ce-8768-d6ec3bd47a96\") "
Sep 15 16:50:17.379672 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.379632    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swhhj\" (UniqueName: \"kubernetes.io/projected/da838de0-317f-41ce-8768-d6ec3bd47a96-kube-api-access-swhhj\") pod \"da838de0-317f-41ce-8768-d6ec3bd47a96\" (UID: \"da838de0-317f-41ce-8768-d6ec3bd47a96\") "
Sep 15 16:50:17.379861 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.379688    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da838de0-317f-41ce-8768-d6ec3bd47a96-kube-rbac-proxy-sar-config\") pod \"da838de0-317f-41ce-8768-d6ec3bd47a96\" (UID: \"da838de0-317f-41ce-8768-d6ec3bd47a96\") "
Sep 15 16:50:17.380110 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.380080    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da838de0-317f-41ce-8768-d6ec3bd47a96-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "da838de0-317f-41ce-8768-d6ec3bd47a96" (UID: "da838de0-317f-41ce-8768-d6ec3bd47a96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:50:17.381645 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.381627    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da838de0-317f-41ce-8768-d6ec3bd47a96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da838de0-317f-41ce-8768-d6ec3bd47a96" (UID: "da838de0-317f-41ce-8768-d6ec3bd47a96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:50:17.381777 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.381755    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da838de0-317f-41ce-8768-d6ec3bd47a96-kube-api-access-swhhj" (OuterVolumeSpecName: "kube-api-access-swhhj") pod "da838de0-317f-41ce-8768-d6ec3bd47a96" (UID: "da838de0-317f-41ce-8768-d6ec3bd47a96"). InnerVolumeSpecName "kube-api-access-swhhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:50:17.480442 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.480416    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da838de0-317f-41ce-8768-d6ec3bd47a96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:50:17.480442 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.480440    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da838de0-317f-41ce-8768-d6ec3bd47a96-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:50:17.480567 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.480457    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swhhj\" (UniqueName: \"kubernetes.io/projected/da838de0-317f-41ce-8768-d6ec3bd47a96-kube-api-access-swhhj\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:50:17.499961 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.499927    2565 generic.go:356] "Generic (PLEG): container finished" podID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerID="76cfd594e2d4f24a5762fffd017e05ef01851fc77ae5f43f2ef3fbec439e442e" exitCode=0
Sep 15 16:50:17.500336 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.499965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" event={"ID":"da838de0-317f-41ce-8768-d6ec3bd47a96","Type":"ContainerDied","Data":"76cfd594e2d4f24a5762fffd017e05ef01851fc77ae5f43f2ef3fbec439e442e"}
Sep 15 16:50:17.500336 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.499984    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"
Sep 15 16:50:17.500336 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.500019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r" event={"ID":"da838de0-317f-41ce-8768-d6ec3bd47a96","Type":"ContainerDied","Data":"dd6bb4fb751189373e2a1b28518110f89a94b9ab0e1ed53b30c9ced4f38cea62"}
Sep 15 16:50:17.500336 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.500048    2565 scope.go:117] "RemoveContainer" containerID="1851b1e9c6caa8241759d33d31086bdffaa73a5b182880bb44f00f16f663e349"
Sep 15 16:50:17.501887 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.501854    2565 generic.go:356] "Generic (PLEG): container finished" podID="a09d582c-d45e-438b-b43c-14735137e8f1" containerID="44f4cbaa4d2bd6e988798d55ba6159422f7f07c3d4b871ee62b14c643d53ef0d" exitCode=0
Sep 15 16:50:17.501995 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.501900    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"
Sep 15 16:50:17.501995 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.501921    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" event={"ID":"a09d582c-d45e-438b-b43c-14735137e8f1","Type":"ContainerDied","Data":"44f4cbaa4d2bd6e988798d55ba6159422f7f07c3d4b871ee62b14c643d53ef0d"}
Sep 15 16:50:17.501995 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.501970    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9" event={"ID":"a09d582c-d45e-438b-b43c-14735137e8f1","Type":"ContainerDied","Data":"5ca9c1f9b3b52ba75e25170f278a0df10469e3dabe56d2ea5962705080d68b75"}
Sep 15 16:50:17.508414 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.508387    2565 scope.go:117] "RemoveContainer" containerID="76cfd594e2d4f24a5762fffd017e05ef01851fc77ae5f43f2ef3fbec439e442e"
Sep 15 16:50:17.515409 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.515390    2565 scope.go:117] "RemoveContainer" containerID="1851b1e9c6caa8241759d33d31086bdffaa73a5b182880bb44f00f16f663e349"
Sep 15 16:50:17.515696 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:17.515675    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1851b1e9c6caa8241759d33d31086bdffaa73a5b182880bb44f00f16f663e349\": container with ID starting with 1851b1e9c6caa8241759d33d31086bdffaa73a5b182880bb44f00f16f663e349 not found: ID does not exist" containerID="1851b1e9c6caa8241759d33d31086bdffaa73a5b182880bb44f00f16f663e349"
Sep 15 16:50:17.515788 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.515702    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1851b1e9c6caa8241759d33d31086bdffaa73a5b182880bb44f00f16f663e349"} err="failed to get container status \"1851b1e9c6caa8241759d33d31086bdffaa73a5b182880bb44f00f16f663e349\": rpc error: code = NotFound desc = could not find container \"1851b1e9c6caa8241759d33d31086bdffaa73a5b182880bb44f00f16f663e349\": container with ID starting with 1851b1e9c6caa8241759d33d31086bdffaa73a5b182880bb44f00f16f663e349 not found: ID does not exist"
Sep 15 16:50:17.515788 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.515732    2565 scope.go:117] "RemoveContainer" containerID="76cfd594e2d4f24a5762fffd017e05ef01851fc77ae5f43f2ef3fbec439e442e"
Sep 15 16:50:17.515991 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:17.515973    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76cfd594e2d4f24a5762fffd017e05ef01851fc77ae5f43f2ef3fbec439e442e\": container with ID starting with 76cfd594e2d4f24a5762fffd017e05ef01851fc77ae5f43f2ef3fbec439e442e not found: ID does not exist" containerID="76cfd594e2d4f24a5762fffd017e05ef01851fc77ae5f43f2ef3fbec439e442e"
Sep 15 16:50:17.516038 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.515997    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76cfd594e2d4f24a5762fffd017e05ef01851fc77ae5f43f2ef3fbec439e442e"} err="failed to get container status \"76cfd594e2d4f24a5762fffd017e05ef01851fc77ae5f43f2ef3fbec439e442e\": rpc error: code = NotFound desc = could not find container \"76cfd594e2d4f24a5762fffd017e05ef01851fc77ae5f43f2ef3fbec439e442e\": container with ID starting with 76cfd594e2d4f24a5762fffd017e05ef01851fc77ae5f43f2ef3fbec439e442e not found: ID does not exist"
Sep 15 16:50:17.516038 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.516011    2565 scope.go:117] "RemoveContainer" containerID="a4fccf354746614691ed8acd3bbc828f05342f1cb8f0604efbd2779995b2b2fd"
Sep 15 16:50:17.517008 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.516914    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"]
Sep 15 16:50:17.520276 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.520257    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daa19-predictor-64588c679c-fz4x9"]
Sep 15 16:50:17.523306 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.523291    2565 scope.go:117] "RemoveContainer" containerID="44f4cbaa4d2bd6e988798d55ba6159422f7f07c3d4b871ee62b14c643d53ef0d"
Sep 15 16:50:17.530652 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.530636    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"]
Sep 15 16:50:17.530700 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.530671    2565 scope.go:117] "RemoveContainer" containerID="a4fccf354746614691ed8acd3bbc828f05342f1cb8f0604efbd2779995b2b2fd"
Sep 15 16:50:17.530983 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:17.530965    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fccf354746614691ed8acd3bbc828f05342f1cb8f0604efbd2779995b2b2fd\": container with ID starting with a4fccf354746614691ed8acd3bbc828f05342f1cb8f0604efbd2779995b2b2fd not found: ID does not exist" containerID="a4fccf354746614691ed8acd3bbc828f05342f1cb8f0604efbd2779995b2b2fd"
Sep 15 16:50:17.531047 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.530994    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fccf354746614691ed8acd3bbc828f05342f1cb8f0604efbd2779995b2b2fd"} err="failed to get container status \"a4fccf354746614691ed8acd3bbc828f05342f1cb8f0604efbd2779995b2b2fd\": rpc error: code = NotFound desc = could not find container \"a4fccf354746614691ed8acd3bbc828f05342f1cb8f0604efbd2779995b2b2fd\": container with ID starting with a4fccf354746614691ed8acd3bbc828f05342f1cb8f0604efbd2779995b2b2fd not found: ID does not exist"
Sep 15 16:50:17.531047 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.531015    2565 scope.go:117] "RemoveContainer" containerID="44f4cbaa4d2bd6e988798d55ba6159422f7f07c3d4b871ee62b14c643d53ef0d"
Sep 15 16:50:17.531298 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:17.531281    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f4cbaa4d2bd6e988798d55ba6159422f7f07c3d4b871ee62b14c643d53ef0d\": container with ID starting with 44f4cbaa4d2bd6e988798d55ba6159422f7f07c3d4b871ee62b14c643d53ef0d not found: ID does not exist" containerID="44f4cbaa4d2bd6e988798d55ba6159422f7f07c3d4b871ee62b14c643d53ef0d"
Sep 15 16:50:17.531363 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.531301    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f4cbaa4d2bd6e988798d55ba6159422f7f07c3d4b871ee62b14c643d53ef0d"} err="failed to get container status \"44f4cbaa4d2bd6e988798d55ba6159422f7f07c3d4b871ee62b14c643d53ef0d\": rpc error: code = NotFound desc = could not find container \"44f4cbaa4d2bd6e988798d55ba6159422f7f07c3d4b871ee62b14c643d53ef0d\": container with ID starting with 44f4cbaa4d2bd6e988798d55ba6159422f7f07c3d4b871ee62b14c643d53ef0d not found: ID does not exist"
Sep 15 16:50:17.534507 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.534480    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daa19-predictor-78598b9488-nx46r"]
Sep 15 16:50:17.549179 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:17.549143    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:19.448468 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:19.448431    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" path="/var/lib/kubelet/pods/a09d582c-d45e-438b-b43c-14735137e8f1/volumes"
Sep 15 16:50:19.448889 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:19.448873    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" path="/var/lib/kubelet/pods/da838de0-317f-41ce-8768-d6ec3bd47a96/volumes"
Sep 15 16:50:21.500242 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:21.500212    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:50:21.500662 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:21.500385    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:50:21.500761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:21.500734    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:50:21.500851 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:21.500821    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:50:23.596595 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:23.596559    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:24.050933 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:24.050888    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:28.619462 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:28.619429    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:28.717418 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:28.717382    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:31.501011 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:31.500977    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:50:31.501011 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:31.500994    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:50:33.667807 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.667767    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:33.747227 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.747196    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"]
Sep 15 16:50:33.747548 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.747518    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kserve-container" containerID="cri-o://f7c8ed9a436f71dbf3a64a50f9308fa0b2e8ba764931b0f0c7a516b39f9f7f50" gracePeriod=30
Sep 15 16:50:33.747684 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.747515    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kube-rbac-proxy" containerID="cri-o://82a86615a51d4edddd417bbd72fba5bb393cc5d3fb1c6366bdce557363e79442" gracePeriod=30
Sep 15 16:50:33.779890 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.779861    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"]
Sep 15 16:50:33.780192 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.780149    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kserve-container" containerID="cri-o://141729a823e41926287923fb090ee3e807fc1cdfc1ca488f29d4b71c72c80da8" gracePeriod=30
Sep 15 16:50:33.780266 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.780225    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kube-rbac-proxy" containerID="cri-o://915e651ce8751c9a1e15bcaf70d74a66649cbb9dc7bfccdbb450c20a2e84ffad" gracePeriod=30
Sep 15 16:50:33.781920 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.781891    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:33.855162 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.855132    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:33.858894 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.858872    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"]
Sep 15 16:50:33.859168 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.859156    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kube-rbac-proxy"
Sep 15 16:50:33.859216 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.859171    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kube-rbac-proxy"
Sep 15 16:50:33.859216 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.859205    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kube-rbac-proxy"
Sep 15 16:50:33.859216 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.859210    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kube-rbac-proxy"
Sep 15 16:50:33.859313 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.859222    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kserve-container"
Sep 15 16:50:33.859313 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.859227    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kserve-container"
Sep 15 16:50:33.859313 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.859234    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kserve-container"
Sep 15 16:50:33.859313 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.859239    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kserve-container"
Sep 15 16:50:33.859313 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.859283    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kserve-container"
Sep 15 16:50:33.859313 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.859289    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kserve-container"
Sep 15 16:50:33.859313 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.859295    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da838de0-317f-41ce-8768-d6ec3bd47a96" containerName="kube-rbac-proxy"
Sep 15 16:50:33.859313 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.859301    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09d582c-d45e-438b-b43c-14735137e8f1" containerName="kube-rbac-proxy"
Sep 15 16:50:33.862086 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.862071    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:33.864597 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.864578    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0591a-kube-rbac-proxy-sar-config\""
Sep 15 16:50:33.864702 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.864578    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0591a-predictor-serving-cert\""
Sep 15 16:50:33.872046 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.872021    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"]
Sep 15 16:50:33.903778 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.903751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0591a-predictor-78848d555b-bpb6k\" (UID: \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\") " pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:33.903881 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.903795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-proxy-tls\") pod \"success-200-isvc-0591a-predictor-78848d555b-bpb6k\" (UID: \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\") " pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:33.903881 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.903865    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7vh9\" (UniqueName: \"kubernetes.io/projected/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-kube-api-access-v7vh9\") pod \"success-200-isvc-0591a-predictor-78848d555b-bpb6k\" (UID: \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\") " pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:33.921124 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.921067    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:33.954915 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.954884    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"]
Sep 15 16:50:33.958206 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.958187    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:33.960806 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.960783    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0591a-predictor-serving-cert\""
Sep 15 16:50:33.960932 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.960843    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0591a-kube-rbac-proxy-sar-config\""
Sep 15 16:50:33.967733 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.967685    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"]
Sep 15 16:50:33.986915 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:33.986888    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:34.004649 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.004618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0591a-predictor-78848d555b-bpb6k\" (UID: \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\") " pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:34.004831 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.004662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-proxy-tls\") pod \"success-200-isvc-0591a-predictor-78848d555b-bpb6k\" (UID: \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\") " pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:34.004831 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.004723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7vh9\" (UniqueName: \"kubernetes.io/projected/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-kube-api-access-v7vh9\") pod \"success-200-isvc-0591a-predictor-78848d555b-bpb6k\" (UID: \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\") " pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:34.004831 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.004755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwc8p\" (UniqueName: \"kubernetes.io/projected/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-kube-api-access-fwc8p\") pod \"error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv\" (UID: \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\") " pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:34.004831 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.004812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv\" (UID: \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\") " pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:34.005100 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.004843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-proxy-tls\") pod \"error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv\" (UID: \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\") " pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:34.005100 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:34.004870    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-0591a-predictor-serving-cert: secret "success-200-isvc-0591a-predictor-serving-cert" not found
Sep 15 16:50:34.005100 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:34.004933    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-proxy-tls podName:898c4f24-2f09-4896-8e8a-7c9ae3f86f6d nodeName:}" failed. No retries permitted until 2026-09-15 16:50:34.504913584 +0000 UTC m=+1425.549299293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-proxy-tls") pod "success-200-isvc-0591a-predictor-78848d555b-bpb6k" (UID: "898c4f24-2f09-4896-8e8a-7c9ae3f86f6d") : secret "success-200-isvc-0591a-predictor-serving-cert" not found
Sep 15 16:50:34.005398 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.005377    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0591a-predictor-78848d555b-bpb6k\" (UID: \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\") " pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:34.015139 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.015112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7vh9\" (UniqueName: \"kubernetes.io/projected/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-kube-api-access-v7vh9\") pod \"success-200-isvc-0591a-predictor-78848d555b-bpb6k\" (UID: \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\") " pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:34.066975 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.066942    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:34.105761 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.105730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-proxy-tls\") pod \"error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv\" (UID: \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\") " pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:34.105913 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.105829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwc8p\" (UniqueName: \"kubernetes.io/projected/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-kube-api-access-fwc8p\") pod \"error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv\" (UID: \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\") " pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:34.105913 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.105879    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv\" (UID: \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\") " pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:34.106463 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.106443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv\" (UID: \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\") " pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:34.108187 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.108168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-proxy-tls\") pod \"error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv\" (UID: \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\") " pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:34.113503 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.113485    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwc8p\" (UniqueName: \"kubernetes.io/projected/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-kube-api-access-fwc8p\") pod \"error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv\" (UID: \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\") " pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:34.168024 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.167992    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:34.269663 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.269578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:34.508366 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.508329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-proxy-tls\") pod \"success-200-isvc-0591a-predictor-78848d555b-bpb6k\" (UID: \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\") " pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:34.510701 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.510682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-proxy-tls\") pod \"success-200-isvc-0591a-predictor-78848d555b-bpb6k\" (UID: \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\") " pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:34.540919 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.540890    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"]
Sep 15 16:50:34.543153 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:50:34.543129    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa57fe9_1d64_4cdd_8f12_c02bac09ada0.slice/crio-73ed7b6148a5e5ea83bdcf3aed24a1a6f0bb3f8de769d8233ce811adf6e39b3d WatchSource:0}: Error finding container 73ed7b6148a5e5ea83bdcf3aed24a1a6f0bb3f8de769d8233ce811adf6e39b3d: Status 404 returned error can't find the container with id 73ed7b6148a5e5ea83bdcf3aed24a1a6f0bb3f8de769d8233ce811adf6e39b3d
Sep 15 16:50:34.567497 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.567471    2565 generic.go:356] "Generic (PLEG): container finished" podID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerID="82a86615a51d4edddd417bbd72fba5bb393cc5d3fb1c6366bdce557363e79442" exitCode=2
Sep 15 16:50:34.567595 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.567540    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" event={"ID":"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925","Type":"ContainerDied","Data":"82a86615a51d4edddd417bbd72fba5bb393cc5d3fb1c6366bdce557363e79442"}
Sep 15 16:50:34.568698 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.568678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" event={"ID":"caa57fe9-1d64-4cdd-8f12-c02bac09ada0","Type":"ContainerStarted","Data":"73ed7b6148a5e5ea83bdcf3aed24a1a6f0bb3f8de769d8233ce811adf6e39b3d"}
Sep 15 16:50:34.570371 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.570353    2565 generic.go:356] "Generic (PLEG): container finished" podID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerID="915e651ce8751c9a1e15bcaf70d74a66649cbb9dc7bfccdbb450c20a2e84ffad" exitCode=2
Sep 15 16:50:34.570443 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.570387    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" event={"ID":"5e683e37-dc5f-459b-be23-2b621ad83e20","Type":"ContainerDied","Data":"915e651ce8751c9a1e15bcaf70d74a66649cbb9dc7bfccdbb450c20a2e84ffad"}
Sep 15 16:50:34.772632 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:34.772593    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:35.041071 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:35.041042    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"]
Sep 15 16:50:35.043539 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:50:35.043510    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898c4f24_2f09_4896_8e8a_7c9ae3f86f6d.slice/crio-f2482d4652521330ddd483db0392987e77456f13ac5160979711b6804752308e WatchSource:0}: Error finding container f2482d4652521330ddd483db0392987e77456f13ac5160979711b6804752308e: Status 404 returned error can't find the container with id f2482d4652521330ddd483db0392987e77456f13ac5160979711b6804752308e
Sep 15 16:50:35.577051 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:35.577009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" event={"ID":"caa57fe9-1d64-4cdd-8f12-c02bac09ada0","Type":"ContainerStarted","Data":"3292aadd89878676ac294c6a3ce115b33068041a7b4bfd2a8a5b0fc3ec4a26da"}
Sep 15 16:50:35.577252 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:35.577080    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" event={"ID":"caa57fe9-1d64-4cdd-8f12-c02bac09ada0","Type":"ContainerStarted","Data":"0eb79e5c06bd5b007b423db32cf134d969fd8012c904942082c407882df841b0"}
Sep 15 16:50:35.577252 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:35.577128    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:35.578935 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:35.578907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" event={"ID":"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d","Type":"ContainerStarted","Data":"d24622ef4726385a4d54486e0e00d4f2a6fe061d3841808ef5d68268fe956e53"}
Sep 15 16:50:35.579075 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:35.578948    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" event={"ID":"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d","Type":"ContainerStarted","Data":"88956f7455403d66291da57d410343c3dc8b80dded5293554c10284fa13f50eb"}
Sep 15 16:50:35.579075 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:35.578978    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" event={"ID":"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d","Type":"ContainerStarted","Data":"f2482d4652521330ddd483db0392987e77456f13ac5160979711b6804752308e"}
Sep 15 16:50:35.579075 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:35.579001    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:35.579075 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:35.579021    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:35.580349 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:35.580324    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:50:35.594540 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:35.594492    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" podStartSLOduration=2.5944793109999997 podStartE2EDuration="2.594479311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:50:35.593847022 +0000 UTC m=+1426.638232753" watchObservedRunningTime="2026-09-15 16:50:35.594479311 +0000 UTC m=+1426.638865042"
Sep 15 16:50:35.610867 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:35.610820    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" podStartSLOduration=2.610807253 podStartE2EDuration="2.610807253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:50:35.60987139 +0000 UTC m=+1426.654257122" watchObservedRunningTime="2026-09-15 16:50:35.610807253 +0000 UTC m=+1426.655192984"
Sep 15 16:50:36.037598 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:36.037552    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:50:36.296443 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:36.296358    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 15 16:50:36.296587 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:36.296358    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 15 16:50:36.584007 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:36.583908    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:50:36.584156 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:36.584104    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:36.585278 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:36.585249    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:50:37.327016 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.326991    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:50:37.432395 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.432359    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e683e37-dc5f-459b-be23-2b621ad83e20-proxy-tls\") pod \"5e683e37-dc5f-459b-be23-2b621ad83e20\" (UID: \"5e683e37-dc5f-459b-be23-2b621ad83e20\") "
Sep 15 16:50:37.432551 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.432418    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gfgh\" (UniqueName: \"kubernetes.io/projected/5e683e37-dc5f-459b-be23-2b621ad83e20-kube-api-access-6gfgh\") pod \"5e683e37-dc5f-459b-be23-2b621ad83e20\" (UID: \"5e683e37-dc5f-459b-be23-2b621ad83e20\") "
Sep 15 16:50:37.432551 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.432517    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e683e37-dc5f-459b-be23-2b621ad83e20-kube-rbac-proxy-sar-config\") pod \"5e683e37-dc5f-459b-be23-2b621ad83e20\" (UID: \"5e683e37-dc5f-459b-be23-2b621ad83e20\") "
Sep 15 16:50:37.433091 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.433056    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e683e37-dc5f-459b-be23-2b621ad83e20-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5e683e37-dc5f-459b-be23-2b621ad83e20" (UID: "5e683e37-dc5f-459b-be23-2b621ad83e20"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:50:37.435111 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.435066    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e683e37-dc5f-459b-be23-2b621ad83e20-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e683e37-dc5f-459b-be23-2b621ad83e20" (UID: "5e683e37-dc5f-459b-be23-2b621ad83e20"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:50:37.435358 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.435329    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e683e37-dc5f-459b-be23-2b621ad83e20-kube-api-access-6gfgh" (OuterVolumeSpecName: "kube-api-access-6gfgh") pod "5e683e37-dc5f-459b-be23-2b621ad83e20" (UID: "5e683e37-dc5f-459b-be23-2b621ad83e20"). InnerVolumeSpecName "kube-api-access-6gfgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:50:37.473779 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.473755    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:50:37.533950 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.533914    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-proxy-tls\") pod \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\" (UID: \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\") "
Sep 15 16:50:37.534137 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.533968    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt9vh\" (UniqueName: \"kubernetes.io/projected/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-kube-api-access-rt9vh\") pod \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\" (UID: \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\") "
Sep 15 16:50:37.534137 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.534008    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-kube-rbac-proxy-sar-config\") pod \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\" (UID: \"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925\") "
Sep 15 16:50:37.534251 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.534192    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e683e37-dc5f-459b-be23-2b621ad83e20-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:50:37.534251 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.534209    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e683e37-dc5f-459b-be23-2b621ad83e20-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:50:37.534251 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.534226    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gfgh\" (UniqueName: \"kubernetes.io/projected/5e683e37-dc5f-459b-be23-2b621ad83e20-kube-api-access-6gfgh\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:50:37.534497 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.534466    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" (UID: "dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:50:37.536635 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.536596    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" (UID: "dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:50:37.536825 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.536798    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-kube-api-access-rt9vh" (OuterVolumeSpecName: "kube-api-access-rt9vh") pod "dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" (UID: "dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925"). InnerVolumeSpecName "kube-api-access-rt9vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:50:37.588690 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.588656    2565 generic.go:356] "Generic (PLEG): container finished" podID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerID="f7c8ed9a436f71dbf3a64a50f9308fa0b2e8ba764931b0f0c7a516b39f9f7f50" exitCode=0
Sep 15 16:50:37.588843 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.588727    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"
Sep 15 16:50:37.588843 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.588754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" event={"ID":"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925","Type":"ContainerDied","Data":"f7c8ed9a436f71dbf3a64a50f9308fa0b2e8ba764931b0f0c7a516b39f9f7f50"}
Sep 15 16:50:37.588843 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.588828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8" event={"ID":"dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925","Type":"ContainerDied","Data":"44abaa2203dc4ea58b7e287fcff9650d3d205e9a2a717e404149a1503fae1ba4"}
Sep 15 16:50:37.589001 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.588865    2565 scope.go:117] "RemoveContainer" containerID="82a86615a51d4edddd417bbd72fba5bb393cc5d3fb1c6366bdce557363e79442"
Sep 15 16:50:37.590588 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.590560    2565 generic.go:356] "Generic (PLEG): container finished" podID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerID="141729a823e41926287923fb090ee3e807fc1cdfc1ca488f29d4b71c72c80da8" exitCode=0
Sep 15 16:50:37.590588 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.590587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" event={"ID":"5e683e37-dc5f-459b-be23-2b621ad83e20","Type":"ContainerDied","Data":"141729a823e41926287923fb090ee3e807fc1cdfc1ca488f29d4b71c72c80da8"}
Sep 15 16:50:37.590812 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.590603    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"
Sep 15 16:50:37.590812 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.590638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f" event={"ID":"5e683e37-dc5f-459b-be23-2b621ad83e20","Type":"ContainerDied","Data":"b4d2090c74d0369bf28a776fb37693208e50196bb207ab54db7bf8e43fa0d72d"}
Sep 15 16:50:37.591219 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.591188    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:50:37.597752 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.597729    2565 scope.go:117] "RemoveContainer" containerID="f7c8ed9a436f71dbf3a64a50f9308fa0b2e8ba764931b0f0c7a516b39f9f7f50"
Sep 15 16:50:37.607023 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.607005    2565 scope.go:117] "RemoveContainer" containerID="82a86615a51d4edddd417bbd72fba5bb393cc5d3fb1c6366bdce557363e79442"
Sep 15 16:50:37.607306 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:37.607287    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a86615a51d4edddd417bbd72fba5bb393cc5d3fb1c6366bdce557363e79442\": container with ID starting with 82a86615a51d4edddd417bbd72fba5bb393cc5d3fb1c6366bdce557363e79442 not found: ID does not exist" containerID="82a86615a51d4edddd417bbd72fba5bb393cc5d3fb1c6366bdce557363e79442"
Sep 15 16:50:37.607388 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.607317    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a86615a51d4edddd417bbd72fba5bb393cc5d3fb1c6366bdce557363e79442"} err="failed to get container status \"82a86615a51d4edddd417bbd72fba5bb393cc5d3fb1c6366bdce557363e79442\": rpc error: code = NotFound desc = could not find container \"82a86615a51d4edddd417bbd72fba5bb393cc5d3fb1c6366bdce557363e79442\": container with ID starting with 82a86615a51d4edddd417bbd72fba5bb393cc5d3fb1c6366bdce557363e79442 not found: ID does not exist"
Sep 15 16:50:37.607388 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.607341    2565 scope.go:117] "RemoveContainer" containerID="f7c8ed9a436f71dbf3a64a50f9308fa0b2e8ba764931b0f0c7a516b39f9f7f50"
Sep 15 16:50:37.607605 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:37.607589    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7c8ed9a436f71dbf3a64a50f9308fa0b2e8ba764931b0f0c7a516b39f9f7f50\": container with ID starting with f7c8ed9a436f71dbf3a64a50f9308fa0b2e8ba764931b0f0c7a516b39f9f7f50 not found: ID does not exist" containerID="f7c8ed9a436f71dbf3a64a50f9308fa0b2e8ba764931b0f0c7a516b39f9f7f50"
Sep 15 16:50:37.607644 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.607611    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7c8ed9a436f71dbf3a64a50f9308fa0b2e8ba764931b0f0c7a516b39f9f7f50"} err="failed to get container status \"f7c8ed9a436f71dbf3a64a50f9308fa0b2e8ba764931b0f0c7a516b39f9f7f50\": rpc error: code = NotFound desc = could not find container \"f7c8ed9a436f71dbf3a64a50f9308fa0b2e8ba764931b0f0c7a516b39f9f7f50\": container with ID starting with f7c8ed9a436f71dbf3a64a50f9308fa0b2e8ba764931b0f0c7a516b39f9f7f50 not found: ID does not exist"
Sep 15 16:50:37.607644 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.607624    2565 scope.go:117] "RemoveContainer" containerID="915e651ce8751c9a1e15bcaf70d74a66649cbb9dc7bfccdbb450c20a2e84ffad"
Sep 15 16:50:37.610870 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.610849    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"]
Sep 15 16:50:37.614243 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.614219    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcea7-predictor-6bfb44d5b-5wgj8"]
Sep 15 16:50:37.615096 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.615080    2565 scope.go:117] "RemoveContainer" containerID="141729a823e41926287923fb090ee3e807fc1cdfc1ca488f29d4b71c72c80da8"
Sep 15 16:50:37.622023 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.622007    2565 scope.go:117] "RemoveContainer" containerID="915e651ce8751c9a1e15bcaf70d74a66649cbb9dc7bfccdbb450c20a2e84ffad"
Sep 15 16:50:37.622263 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:37.622245    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915e651ce8751c9a1e15bcaf70d74a66649cbb9dc7bfccdbb450c20a2e84ffad\": container with ID starting with 915e651ce8751c9a1e15bcaf70d74a66649cbb9dc7bfccdbb450c20a2e84ffad not found: ID does not exist" containerID="915e651ce8751c9a1e15bcaf70d74a66649cbb9dc7bfccdbb450c20a2e84ffad"
Sep 15 16:50:37.622309 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.622269    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915e651ce8751c9a1e15bcaf70d74a66649cbb9dc7bfccdbb450c20a2e84ffad"} err="failed to get container status \"915e651ce8751c9a1e15bcaf70d74a66649cbb9dc7bfccdbb450c20a2e84ffad\": rpc error: code = NotFound desc = could not find container \"915e651ce8751c9a1e15bcaf70d74a66649cbb9dc7bfccdbb450c20a2e84ffad\": container with ID starting with 915e651ce8751c9a1e15bcaf70d74a66649cbb9dc7bfccdbb450c20a2e84ffad not found: ID does not exist"
Sep 15 16:50:37.622309 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.622284    2565 scope.go:117] "RemoveContainer" containerID="141729a823e41926287923fb090ee3e807fc1cdfc1ca488f29d4b71c72c80da8"
Sep 15 16:50:37.622607 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:50:37.622589    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141729a823e41926287923fb090ee3e807fc1cdfc1ca488f29d4b71c72c80da8\": container with ID starting with 141729a823e41926287923fb090ee3e807fc1cdfc1ca488f29d4b71c72c80da8 not found: ID does not exist" containerID="141729a823e41926287923fb090ee3e807fc1cdfc1ca488f29d4b71c72c80da8"
Sep 15 16:50:37.622677 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.622610    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141729a823e41926287923fb090ee3e807fc1cdfc1ca488f29d4b71c72c80da8"} err="failed to get container status \"141729a823e41926287923fb090ee3e807fc1cdfc1ca488f29d4b71c72c80da8\": rpc error: code = NotFound desc = could not find container \"141729a823e41926287923fb090ee3e807fc1cdfc1ca488f29d4b71c72c80da8\": container with ID starting with 141729a823e41926287923fb090ee3e807fc1cdfc1ca488f29d4b71c72c80da8 not found: ID does not exist"
Sep 15 16:50:37.624387 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.624367    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"]
Sep 15 16:50:37.627928 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.627907    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcea7-predictor-597cb8b869-vsl4f"]
Sep 15 16:50:37.635234 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.635208    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:50:37.635234 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.635237    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt9vh\" (UniqueName: \"kubernetes.io/projected/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-kube-api-access-rt9vh\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:50:37.635407 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:37.635252    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:50:39.448393 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:39.448360    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" path="/var/lib/kubelet/pods/5e683e37-dc5f-459b-be23-2b621ad83e20/volumes"
Sep 15 16:50:39.448815 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:39.448799    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" path="/var/lib/kubelet/pods/dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925/volumes"
Sep 15 16:50:41.501405 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:41.501368    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:50:41.501405 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:41.501389    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:50:41.588685 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:41.588652    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:50:41.589311 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:41.589280    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:50:42.595347 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:42.595276    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:50:42.595921 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:42.595895    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:50:51.501223 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:51.501180    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:50:51.501223 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:51.501207    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:50:51.589963 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:51.589924    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:50:52.596087 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:50:52.596044    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:51:01.501628 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:01.501601    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:51:01.502063 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:01.501679    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:51:01.589340 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:01.589293    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:51:02.596670 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:02.596627    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:51:11.590091 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:11.590046    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:51:12.596076 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:12.596032    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:51:13.510250 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:13.510216    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:51:13.609344 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:13.609313    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:51:21.589968 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:21.589927    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:51:22.597112 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:22.597083    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 16:51:23.612585 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.612554    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:51:23.674492 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.674454    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"]
Sep 15 16:51:23.674726 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.674680    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kserve-container" containerID="cri-o://13e1c2c136a9211af88c10e5a5526b22ad2100fcc48cf08c195027264ef36c77" gracePeriod=30
Sep 15 16:51:23.674817 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.674733    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kube-rbac-proxy" containerID="cri-o://6029a319f09b1a8b720bb55dd73091f332bf1211abbf554c64dc2841f486c495" gracePeriod=30
Sep 15 16:51:23.699126 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.699082    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:51:23.709133 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.709109    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"]
Sep 15 16:51:23.709338 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.709309    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kserve-container" containerID="cri-o://4c1ff538b69ea9ab0dc46c2c8f76233c81e5e4fb813ed2bea26e4128564181c6" gracePeriod=30
Sep 15 16:51:23.709462 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.709347    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kube-rbac-proxy" containerID="cri-o://49329ebe531f8fdd64e0331fcbbf7bd76ce52b2203394ad9ba6e46ab191f455d" gracePeriod=30
Sep 15 16:51:23.781457 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.781426    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:51:23.785722 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.785683    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"]
Sep 15 16:51:23.786042 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.786012    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kserve-container"
Sep 15 16:51:23.786042 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.786034    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kserve-container"
Sep 15 16:51:23.786167 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.786046    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kserve-container"
Sep 15 16:51:23.786167 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.786054    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kserve-container"
Sep 15 16:51:23.786167 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.786077    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kube-rbac-proxy"
Sep 15 16:51:23.786167 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.786082    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kube-rbac-proxy"
Sep 15 16:51:23.786167 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.786088    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kube-rbac-proxy"
Sep 15 16:51:23.786167 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.786093    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kube-rbac-proxy"
Sep 15 16:51:23.786167 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.786156    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kserve-container"
Sep 15 16:51:23.786167 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.786165    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd7b2af1-c7d9-4320-bbb7-4b5a5d6d0925" containerName="kube-rbac-proxy"
Sep 15 16:51:23.786396 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.786172    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kube-rbac-proxy"
Sep 15 16:51:23.786396 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.786178    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e683e37-dc5f-459b-be23-2b621ad83e20" containerName="kserve-container"
Sep 15 16:51:23.788977 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.788959    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:23.791359 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.791336    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9d406-predictor-serving-cert\""
Sep 15 16:51:23.791657 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.791640    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9d406-kube-rbac-proxy-sar-config\""
Sep 15 16:51:23.797234 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.797212    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"]
Sep 15 16:51:23.842055 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.842023    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:51:23.877911 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.877828    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"]
Sep 15 16:51:23.881645 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.881626    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:23.884377 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.884357    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9d406-kube-rbac-proxy-sar-config\""
Sep 15 16:51:23.884505 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.884360    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9d406-predictor-serving-cert\""
Sep 15 16:51:23.890218 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.890192    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"]
Sep 15 16:51:23.892160 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.892127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7564c558-9f3b-4f34-b01f-5051282ddc26-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9d406-predictor-9dcd77c94-qkb6v\" (UID: \"7564c558-9f3b-4f34-b01f-5051282ddc26\") " pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:23.892269 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.892174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7564c558-9f3b-4f34-b01f-5051282ddc26-proxy-tls\") pod \"success-200-isvc-9d406-predictor-9dcd77c94-qkb6v\" (UID: \"7564c558-9f3b-4f34-b01f-5051282ddc26\") " pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:23.892316 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.892276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxbn\" (UniqueName: \"kubernetes.io/projected/7564c558-9f3b-4f34-b01f-5051282ddc26-kube-api-access-qdxbn\") pod \"success-200-isvc-9d406-predictor-9dcd77c94-qkb6v\" (UID: \"7564c558-9f3b-4f34-b01f-5051282ddc26\") " pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:23.941932 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.941900    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:51:23.992679 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.992653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8323660f-7c26-4c6a-a210-0cf03e067abe-proxy-tls\") pod \"error-404-isvc-9d406-predictor-c94cb4b67-9qvp9\" (UID: \"8323660f-7c26-4c6a-a210-0cf03e067abe\") " pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:23.992816 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.992688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7564c558-9f3b-4f34-b01f-5051282ddc26-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9d406-predictor-9dcd77c94-qkb6v\" (UID: \"7564c558-9f3b-4f34-b01f-5051282ddc26\") " pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:23.992816 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.992739    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8323660f-7c26-4c6a-a210-0cf03e067abe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9d406-predictor-c94cb4b67-9qvp9\" (UID: \"8323660f-7c26-4c6a-a210-0cf03e067abe\") " pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:23.992816 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.992785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7564c558-9f3b-4f34-b01f-5051282ddc26-proxy-tls\") pod \"success-200-isvc-9d406-predictor-9dcd77c94-qkb6v\" (UID: \"7564c558-9f3b-4f34-b01f-5051282ddc26\") " pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:23.992932 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.992915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxbn\" (UniqueName: \"kubernetes.io/projected/7564c558-9f3b-4f34-b01f-5051282ddc26-kube-api-access-qdxbn\") pod \"success-200-isvc-9d406-predictor-9dcd77c94-qkb6v\" (UID: \"7564c558-9f3b-4f34-b01f-5051282ddc26\") " pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:23.992981 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.992966    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f97sj\" (UniqueName: \"kubernetes.io/projected/8323660f-7c26-4c6a-a210-0cf03e067abe-kube-api-access-f97sj\") pod \"error-404-isvc-9d406-predictor-c94cb4b67-9qvp9\" (UID: \"8323660f-7c26-4c6a-a210-0cf03e067abe\") " pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:23.993444 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.993420    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7564c558-9f3b-4f34-b01f-5051282ddc26-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9d406-predictor-9dcd77c94-qkb6v\" (UID: \"7564c558-9f3b-4f34-b01f-5051282ddc26\") " pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:23.995154 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:23.995136    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7564c558-9f3b-4f34-b01f-5051282ddc26-proxy-tls\") pod \"success-200-isvc-9d406-predictor-9dcd77c94-qkb6v\" (UID: \"7564c558-9f3b-4f34-b01f-5051282ddc26\") " pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:24.000932 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.000909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxbn\" (UniqueName: \"kubernetes.io/projected/7564c558-9f3b-4f34-b01f-5051282ddc26-kube-api-access-qdxbn\") pod \"success-200-isvc-9d406-predictor-9dcd77c94-qkb6v\" (UID: \"7564c558-9f3b-4f34-b01f-5051282ddc26\") " pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:24.042858 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.042833    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:51:24.093724 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.093681    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f97sj\" (UniqueName: \"kubernetes.io/projected/8323660f-7c26-4c6a-a210-0cf03e067abe-kube-api-access-f97sj\") pod \"error-404-isvc-9d406-predictor-c94cb4b67-9qvp9\" (UID: \"8323660f-7c26-4c6a-a210-0cf03e067abe\") " pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:24.093835 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.093735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8323660f-7c26-4c6a-a210-0cf03e067abe-proxy-tls\") pod \"error-404-isvc-9d406-predictor-c94cb4b67-9qvp9\" (UID: \"8323660f-7c26-4c6a-a210-0cf03e067abe\") " pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:24.093835 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.093830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8323660f-7c26-4c6a-a210-0cf03e067abe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9d406-predictor-c94cb4b67-9qvp9\" (UID: \"8323660f-7c26-4c6a-a210-0cf03e067abe\") " pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:24.094374 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.094348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8323660f-7c26-4c6a-a210-0cf03e067abe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9d406-predictor-c94cb4b67-9qvp9\" (UID: \"8323660f-7c26-4c6a-a210-0cf03e067abe\") " pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:24.096170 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.096148    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8323660f-7c26-4c6a-a210-0cf03e067abe-proxy-tls\") pod \"error-404-isvc-9d406-predictor-c94cb4b67-9qvp9\" (UID: \"8323660f-7c26-4c6a-a210-0cf03e067abe\") " pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:24.100655 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.100623    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:24.101454 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.101433    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f97sj\" (UniqueName: \"kubernetes.io/projected/8323660f-7c26-4c6a-a210-0cf03e067abe-kube-api-access-f97sj\") pod \"error-404-isvc-9d406-predictor-c94cb4b67-9qvp9\" (UID: \"8323660f-7c26-4c6a-a210-0cf03e067abe\") " pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:24.193662 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.193629    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:24.372882 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.372854    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"]
Sep 15 16:51:24.375310 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:51:24.375285    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7564c558_9f3b_4f34_b01f_5051282ddc26.slice/crio-37b344ec8c2de253050e240d08f1891613f8b1f8be88c0e64e361034edf3d6c2 WatchSource:0}: Error finding container 37b344ec8c2de253050e240d08f1891613f8b1f8be88c0e64e361034edf3d6c2: Status 404 returned error can't find the container with id 37b344ec8c2de253050e240d08f1891613f8b1f8be88c0e64e361034edf3d6c2
Sep 15 16:51:24.464651 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.464629    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"]
Sep 15 16:51:24.467014 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:51:24.466991    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8323660f_7c26_4c6a_a210_0cf03e067abe.slice/crio-886c9b628c5476bcb9b0e723ee68e62a29feaf28d58e012bf0e6f2860ebe211e WatchSource:0}: Error finding container 886c9b628c5476bcb9b0e723ee68e62a29feaf28d58e012bf0e6f2860ebe211e: Status 404 returned error can't find the container with id 886c9b628c5476bcb9b0e723ee68e62a29feaf28d58e012bf0e6f2860ebe211e
Sep 15 16:51:24.770395 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.770305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" event={"ID":"7564c558-9f3b-4f34-b01f-5051282ddc26","Type":"ContainerStarted","Data":"819320e70a2a52b76f5ff712c471cc77167a6d14dc39723b42455b4a28934e75"}
Sep 15 16:51:24.770395 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.770371    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" event={"ID":"7564c558-9f3b-4f34-b01f-5051282ddc26","Type":"ContainerStarted","Data":"449532a53b02c756d34d9d11af26e0cb3fe1f559d084f67713c17f9dea6f40f2"}
Sep 15 16:51:24.770914 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.770402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" event={"ID":"7564c558-9f3b-4f34-b01f-5051282ddc26","Type":"ContainerStarted","Data":"37b344ec8c2de253050e240d08f1891613f8b1f8be88c0e64e361034edf3d6c2"}
Sep 15 16:51:24.770914 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.770439    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:24.772218 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.772193    2565 generic.go:356] "Generic (PLEG): container finished" podID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerID="6029a319f09b1a8b720bb55dd73091f332bf1211abbf554c64dc2841f486c495" exitCode=2
Sep 15 16:51:24.772356 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.772267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" event={"ID":"90c20cef-c135-4127-9b1b-fbf02e97ad25","Type":"ContainerDied","Data":"6029a319f09b1a8b720bb55dd73091f332bf1211abbf554c64dc2841f486c495"}
Sep 15 16:51:24.774187 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.774165    2565 generic.go:356] "Generic (PLEG): container finished" podID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerID="49329ebe531f8fdd64e0331fcbbf7bd76ce52b2203394ad9ba6e46ab191f455d" exitCode=2
Sep 15 16:51:24.774301 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.774240    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" event={"ID":"e6e325ea-ffa7-42b0-ab3a-c5135cb72073","Type":"ContainerDied","Data":"49329ebe531f8fdd64e0331fcbbf7bd76ce52b2203394ad9ba6e46ab191f455d"}
Sep 15 16:51:24.776101 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.776077    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" event={"ID":"8323660f-7c26-4c6a-a210-0cf03e067abe","Type":"ContainerStarted","Data":"556638e337a0c99d31bc5074da1e4ad457758594f3aa5cb37464e64a74e8570f"}
Sep 15 16:51:24.776191 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.776134    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" event={"ID":"8323660f-7c26-4c6a-a210-0cf03e067abe","Type":"ContainerStarted","Data":"73f7552e5a6483062937b2ec8b272b9ef0fcee6764fc0da6f25ec0b96eaf00ca"}
Sep 15 16:51:24.776191 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.776171    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:24.776297 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.776208    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:24.776297 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.776237    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" event={"ID":"8323660f-7c26-4c6a-a210-0cf03e067abe","Type":"ContainerStarted","Data":"886c9b628c5476bcb9b0e723ee68e62a29feaf28d58e012bf0e6f2860ebe211e"}
Sep 15 16:51:24.777499 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.777476    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:51:24.790299 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.790259    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" podStartSLOduration=1.7902460169999999 podStartE2EDuration="1.790246017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:51:24.789773703 +0000 UTC m=+1475.834159435" watchObservedRunningTime="2026-09-15 16:51:24.790246017 +0000 UTC m=+1475.834631749"
Sep 15 16:51:24.806933 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:24.806893    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" podStartSLOduration=1.80688223 podStartE2EDuration="1.80688223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:51:24.805546658 +0000 UTC m=+1475.849932392" watchObservedRunningTime="2026-09-15 16:51:24.80688223 +0000 UTC m=+1475.851267961"
Sep 15 16:51:25.780565 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:25.780522    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:51:25.781050 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:25.780634    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:25.781835 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:25.781799    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:51:26.495882 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:26.495835    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 15 16:51:26.495882 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:26.495850    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 15 16:51:26.784949 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:26.784909    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:51:26.929695 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:26.929667    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:51:27.018179 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.018102    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90c20cef-c135-4127-9b1b-fbf02e97ad25-proxy-tls\") pod \"90c20cef-c135-4127-9b1b-fbf02e97ad25\" (UID: \"90c20cef-c135-4127-9b1b-fbf02e97ad25\") "
Sep 15 16:51:27.018325 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.018193    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90c20cef-c135-4127-9b1b-fbf02e97ad25-kube-rbac-proxy-sar-config\") pod \"90c20cef-c135-4127-9b1b-fbf02e97ad25\" (UID: \"90c20cef-c135-4127-9b1b-fbf02e97ad25\") "
Sep 15 16:51:27.018325 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.018240    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxp9k\" (UniqueName: \"kubernetes.io/projected/90c20cef-c135-4127-9b1b-fbf02e97ad25-kube-api-access-dxp9k\") pod \"90c20cef-c135-4127-9b1b-fbf02e97ad25\" (UID: \"90c20cef-c135-4127-9b1b-fbf02e97ad25\") "
Sep 15 16:51:27.018546 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.018518    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c20cef-c135-4127-9b1b-fbf02e97ad25-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "90c20cef-c135-4127-9b1b-fbf02e97ad25" (UID: "90c20cef-c135-4127-9b1b-fbf02e97ad25"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:51:27.021822 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.021795    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c20cef-c135-4127-9b1b-fbf02e97ad25-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "90c20cef-c135-4127-9b1b-fbf02e97ad25" (UID: "90c20cef-c135-4127-9b1b-fbf02e97ad25"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:51:27.021914 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.021853    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c20cef-c135-4127-9b1b-fbf02e97ad25-kube-api-access-dxp9k" (OuterVolumeSpecName: "kube-api-access-dxp9k") pod "90c20cef-c135-4127-9b1b-fbf02e97ad25" (UID: "90c20cef-c135-4127-9b1b-fbf02e97ad25"). InnerVolumeSpecName "kube-api-access-dxp9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:51:27.042121 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.042099    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:51:27.119389 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.119354    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9lsn\" (UniqueName: \"kubernetes.io/projected/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-kube-api-access-f9lsn\") pod \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\" (UID: \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\") "
Sep 15 16:51:27.119526 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.119415    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-proxy-tls\") pod \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\" (UID: \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\") "
Sep 15 16:51:27.119526 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.119444    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-kube-rbac-proxy-sar-config\") pod \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\" (UID: \"e6e325ea-ffa7-42b0-ab3a-c5135cb72073\") "
Sep 15 16:51:27.119609 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.119562    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90c20cef-c135-4127-9b1b-fbf02e97ad25-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:51:27.119609 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.119575    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxp9k\" (UniqueName: \"kubernetes.io/projected/90c20cef-c135-4127-9b1b-fbf02e97ad25-kube-api-access-dxp9k\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:51:27.119609 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.119588    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90c20cef-c135-4127-9b1b-fbf02e97ad25-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:51:27.119889 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.119862    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e6e325ea-ffa7-42b0-ab3a-c5135cb72073" (UID: "e6e325ea-ffa7-42b0-ab3a-c5135cb72073"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:51:27.121587 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.121570    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e6e325ea-ffa7-42b0-ab3a-c5135cb72073" (UID: "e6e325ea-ffa7-42b0-ab3a-c5135cb72073"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:51:27.121646 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.121568    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-kube-api-access-f9lsn" (OuterVolumeSpecName: "kube-api-access-f9lsn") pod "e6e325ea-ffa7-42b0-ab3a-c5135cb72073" (UID: "e6e325ea-ffa7-42b0-ab3a-c5135cb72073"). InnerVolumeSpecName "kube-api-access-f9lsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:51:27.220116 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.220077    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:51:27.220116 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.220111    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:51:27.220116 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.220120    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9lsn\" (UniqueName: \"kubernetes.io/projected/e6e325ea-ffa7-42b0-ab3a-c5135cb72073-kube-api-access-f9lsn\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 16:51:27.790239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.790197    2565 generic.go:356] "Generic (PLEG): container finished" podID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerID="4c1ff538b69ea9ab0dc46c2c8f76233c81e5e4fb813ed2bea26e4128564181c6" exitCode=0
Sep 15 16:51:27.790652 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.790238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" event={"ID":"e6e325ea-ffa7-42b0-ab3a-c5135cb72073","Type":"ContainerDied","Data":"4c1ff538b69ea9ab0dc46c2c8f76233c81e5e4fb813ed2bea26e4128564181c6"}
Sep 15 16:51:27.790652 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.790266    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"
Sep 15 16:51:27.790652 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.790316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm" event={"ID":"e6e325ea-ffa7-42b0-ab3a-c5135cb72073","Type":"ContainerDied","Data":"1a09eebf42af86bde9f12a5b0455a88a11ded7c3405b09c63baf40c277268404"}
Sep 15 16:51:27.790652 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.790357    2565 scope.go:117] "RemoveContainer" containerID="49329ebe531f8fdd64e0331fcbbf7bd76ce52b2203394ad9ba6e46ab191f455d"
Sep 15 16:51:27.792166 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.792142    2565 generic.go:356] "Generic (PLEG): container finished" podID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerID="13e1c2c136a9211af88c10e5a5526b22ad2100fcc48cf08c195027264ef36c77" exitCode=0
Sep 15 16:51:27.792387 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.792189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" event={"ID":"90c20cef-c135-4127-9b1b-fbf02e97ad25","Type":"ContainerDied","Data":"13e1c2c136a9211af88c10e5a5526b22ad2100fcc48cf08c195027264ef36c77"}
Sep 15 16:51:27.792387 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.792238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727" event={"ID":"90c20cef-c135-4127-9b1b-fbf02e97ad25","Type":"ContainerDied","Data":"d0051314872e1811d01db7f2b01bf1bfe3f9f1e2221a0a1f422c091e2124ae7d"}
Sep 15 16:51:27.792387 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.792325    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"
Sep 15 16:51:27.800596 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.800469    2565 scope.go:117] "RemoveContainer" containerID="4c1ff538b69ea9ab0dc46c2c8f76233c81e5e4fb813ed2bea26e4128564181c6"
Sep 15 16:51:27.807885 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.807865    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"]
Sep 15 16:51:27.808182 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.808151    2565 scope.go:117] "RemoveContainer" containerID="49329ebe531f8fdd64e0331fcbbf7bd76ce52b2203394ad9ba6e46ab191f455d"
Sep 15 16:51:27.808690 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:51:27.808662    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49329ebe531f8fdd64e0331fcbbf7bd76ce52b2203394ad9ba6e46ab191f455d\": container with ID starting with 49329ebe531f8fdd64e0331fcbbf7bd76ce52b2203394ad9ba6e46ab191f455d not found: ID does not exist" containerID="49329ebe531f8fdd64e0331fcbbf7bd76ce52b2203394ad9ba6e46ab191f455d"
Sep 15 16:51:27.808821 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.808728    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49329ebe531f8fdd64e0331fcbbf7bd76ce52b2203394ad9ba6e46ab191f455d"} err="failed to get container status \"49329ebe531f8fdd64e0331fcbbf7bd76ce52b2203394ad9ba6e46ab191f455d\": rpc error: code = NotFound desc = could not find container \"49329ebe531f8fdd64e0331fcbbf7bd76ce52b2203394ad9ba6e46ab191f455d\": container with ID starting with 49329ebe531f8fdd64e0331fcbbf7bd76ce52b2203394ad9ba6e46ab191f455d not found: ID does not exist"
Sep 15 16:51:27.808821 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.808751    2565 scope.go:117] "RemoveContainer" containerID="4c1ff538b69ea9ab0dc46c2c8f76233c81e5e4fb813ed2bea26e4128564181c6"
Sep 15 16:51:27.809040 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:51:27.809018    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c1ff538b69ea9ab0dc46c2c8f76233c81e5e4fb813ed2bea26e4128564181c6\": container with ID starting with 4c1ff538b69ea9ab0dc46c2c8f76233c81e5e4fb813ed2bea26e4128564181c6 not found: ID does not exist" containerID="4c1ff538b69ea9ab0dc46c2c8f76233c81e5e4fb813ed2bea26e4128564181c6"
Sep 15 16:51:27.809100 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.809045    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c1ff538b69ea9ab0dc46c2c8f76233c81e5e4fb813ed2bea26e4128564181c6"} err="failed to get container status \"4c1ff538b69ea9ab0dc46c2c8f76233c81e5e4fb813ed2bea26e4128564181c6\": rpc error: code = NotFound desc = could not find container \"4c1ff538b69ea9ab0dc46c2c8f76233c81e5e4fb813ed2bea26e4128564181c6\": container with ID starting with 4c1ff538b69ea9ab0dc46c2c8f76233c81e5e4fb813ed2bea26e4128564181c6 not found: ID does not exist"
Sep 15 16:51:27.809100 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.809061    2565 scope.go:117] "RemoveContainer" containerID="6029a319f09b1a8b720bb55dd73091f332bf1211abbf554c64dc2841f486c495"
Sep 15 16:51:27.810219 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.810201    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1d1fb-predictor-5c99c98b8f-cn2lm"]
Sep 15 16:51:27.815994 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.815978    2565 scope.go:117] "RemoveContainer" containerID="13e1c2c136a9211af88c10e5a5526b22ad2100fcc48cf08c195027264ef36c77"
Sep 15 16:51:27.820253 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.820235    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"]
Sep 15 16:51:27.822873 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.822853    2565 scope.go:117] "RemoveContainer" containerID="6029a319f09b1a8b720bb55dd73091f332bf1211abbf554c64dc2841f486c495"
Sep 15 16:51:27.823116 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:51:27.823097    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6029a319f09b1a8b720bb55dd73091f332bf1211abbf554c64dc2841f486c495\": container with ID starting with 6029a319f09b1a8b720bb55dd73091f332bf1211abbf554c64dc2841f486c495 not found: ID does not exist" containerID="6029a319f09b1a8b720bb55dd73091f332bf1211abbf554c64dc2841f486c495"
Sep 15 16:51:27.823184 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.823124    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6029a319f09b1a8b720bb55dd73091f332bf1211abbf554c64dc2841f486c495"} err="failed to get container status \"6029a319f09b1a8b720bb55dd73091f332bf1211abbf554c64dc2841f486c495\": rpc error: code = NotFound desc = could not find container \"6029a319f09b1a8b720bb55dd73091f332bf1211abbf554c64dc2841f486c495\": container with ID starting with 6029a319f09b1a8b720bb55dd73091f332bf1211abbf554c64dc2841f486c495 not found: ID does not exist"
Sep 15 16:51:27.823184 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.823144    2565 scope.go:117] "RemoveContainer" containerID="13e1c2c136a9211af88c10e5a5526b22ad2100fcc48cf08c195027264ef36c77"
Sep 15 16:51:27.823429 ip-10-0-134-77 kubenswrapper[2565]: E0915 16:51:27.823409    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e1c2c136a9211af88c10e5a5526b22ad2100fcc48cf08c195027264ef36c77\": container with ID starting with 13e1c2c136a9211af88c10e5a5526b22ad2100fcc48cf08c195027264ef36c77 not found: ID does not exist" containerID="13e1c2c136a9211af88c10e5a5526b22ad2100fcc48cf08c195027264ef36c77"
Sep 15 16:51:27.823506 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.823434    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e1c2c136a9211af88c10e5a5526b22ad2100fcc48cf08c195027264ef36c77"} err="failed to get container status \"13e1c2c136a9211af88c10e5a5526b22ad2100fcc48cf08c195027264ef36c77\": rpc error: code = NotFound desc = could not find container \"13e1c2c136a9211af88c10e5a5526b22ad2100fcc48cf08c195027264ef36c77\": container with ID starting with 13e1c2c136a9211af88c10e5a5526b22ad2100fcc48cf08c195027264ef36c77 not found: ID does not exist"
Sep 15 16:51:27.825971 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:27.825949    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1d1fb-predictor-7c46f69ccb-95727"]
Sep 15 16:51:29.450075 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:29.450047    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" path="/var/lib/kubelet/pods/90c20cef-c135-4127-9b1b-fbf02e97ad25/volumes"
Sep 15 16:51:29.450444 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:29.450431    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" path="/var/lib/kubelet/pods/e6e325ea-ffa7-42b0-ab3a-c5135cb72073/volumes"
Sep 15 16:51:30.785411 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:30.785380    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:51:30.785887 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:30.785859    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:51:31.590532 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:31.590502    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 16:51:31.789993 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:31.789965    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:51:31.790488 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:31.790460    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:51:40.786795 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:40.786743    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:51:41.791350 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:41.791307    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:51:45.765908 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:45.765873    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:51:45.825309 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:45.825252    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:51:50.786270 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:50.786228    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:51:51.791353 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:51:51.791315    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:52:00.786399 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:52:00.786356    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:52:01.791265 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:52:01.791227    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:52:10.786530 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:52:10.786499    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 16:52:11.791373 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:52:11.791344    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 16:52:23.823239 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:52:23.823199    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:52:23.953974 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:52:23.953927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:53:15.109834 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:53:15.109800    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:59:30.405785 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:30.405747    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:59:58.505735 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.505629    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:59:58.676280 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.676244    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"]
Sep 15 16:59:58.676593 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.676538    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kserve-container" containerID="cri-o://88956f7455403d66291da57d410343c3dc8b80dded5293554c10284fa13f50eb" gracePeriod=30
Sep 15 16:59:58.676593 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.676569    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kube-rbac-proxy" containerID="cri-o://d24622ef4726385a4d54486e0e00d4f2a6fe061d3841808ef5d68268fe956e53" gracePeriod=30
Sep 15 16:59:58.705039 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.705012    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:59:58.730749 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.730724    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"]
Sep 15 16:59:58.731018 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.731006    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kube-rbac-proxy"
Sep 15 16:59:58.731065 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.731019    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kube-rbac-proxy"
Sep 15 16:59:58.731065 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.731028    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kube-rbac-proxy"
Sep 15 16:59:58.731065 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.731036    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kube-rbac-proxy"
Sep 15 16:59:58.731065 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.731045    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kserve-container"
Sep 15 16:59:58.731065 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.731051    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kserve-container"
Sep 15 16:59:58.731065 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.731063    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kserve-container"
Sep 15 16:59:58.731266 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.731069    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kserve-container"
Sep 15 16:59:58.731266 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.731120    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kserve-container"
Sep 15 16:59:58.731266 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.731129    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kserve-container"
Sep 15 16:59:58.731266 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.731135    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6e325ea-ffa7-42b0-ab3a-c5135cb72073" containerName="kube-rbac-proxy"
Sep 15 16:59:58.731266 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.731142    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c20cef-c135-4127-9b1b-fbf02e97ad25" containerName="kube-rbac-proxy"
Sep 15 16:59:58.734068 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.734053    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 16:59:58.736693 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.736671    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6a778-kube-rbac-proxy-sar-config\""
Sep 15 16:59:58.736928 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.736909    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6a778-predictor-serving-cert\""
Sep 15 16:59:58.742607 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.742582    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"]
Sep 15 16:59:58.783147 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.783065    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"]
Sep 15 16:59:58.783396 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.783360    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kserve-container" containerID="cri-o://0eb79e5c06bd5b007b423db32cf134d969fd8012c904942082c407882df841b0" gracePeriod=30
Sep 15 16:59:58.783396 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.783381    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kube-rbac-proxy" containerID="cri-o://3292aadd89878676ac294c6a3ce115b33068041a7b4bfd2a8a5b0fc3ec4a26da" gracePeriod=30
Sep 15 16:59:58.785958 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.785916    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:59:58.802266 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.802232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0365941-c04c-4a07-bed6-c012cba8189b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6a778-predictor-ff6c4454b-ftnlq\" (UID: \"e0365941-c04c-4a07-bed6-c012cba8189b\") " pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 16:59:58.802381 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.802289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0365941-c04c-4a07-bed6-c012cba8189b-proxy-tls\") pod \"success-200-isvc-6a778-predictor-ff6c4454b-ftnlq\" (UID: \"e0365941-c04c-4a07-bed6-c012cba8189b\") " pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 16:59:58.802381 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.802316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx6g9\" (UniqueName: \"kubernetes.io/projected/e0365941-c04c-4a07-bed6-c012cba8189b-kube-api-access-jx6g9\") pod \"success-200-isvc-6a778-predictor-ff6c4454b-ftnlq\" (UID: \"e0365941-c04c-4a07-bed6-c012cba8189b\") " pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 16:59:58.827418 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.827384    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"]
Sep 15 16:59:58.830665 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.830641    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 16:59:58.833208 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.833187    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6a778-predictor-serving-cert\""
Sep 15 16:59:58.833368 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.833187    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6a778-kube-rbac-proxy-sar-config\""
Sep 15 16:59:58.840227 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.840205    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"]
Sep 15 16:59:58.860926 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.860895    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:59:58.902869 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.902849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04de9312-8a16-4824-ac77-fad3abef545a-proxy-tls\") pod \"error-404-isvc-6a778-predictor-84cc69f4d7-sn28q\" (UID: \"04de9312-8a16-4824-ac77-fad3abef545a\") " pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 16:59:58.902947 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.902885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0365941-c04c-4a07-bed6-c012cba8189b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6a778-predictor-ff6c4454b-ftnlq\" (UID: \"e0365941-c04c-4a07-bed6-c012cba8189b\") " pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 16:59:58.902947 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.902910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzddv\" (UniqueName: \"kubernetes.io/projected/04de9312-8a16-4824-ac77-fad3abef545a-kube-api-access-tzddv\") pod \"error-404-isvc-6a778-predictor-84cc69f4d7-sn28q\" (UID: \"04de9312-8a16-4824-ac77-fad3abef545a\") " pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 16:59:58.903056 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.902959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0365941-c04c-4a07-bed6-c012cba8189b-proxy-tls\") pod \"success-200-isvc-6a778-predictor-ff6c4454b-ftnlq\" (UID: \"e0365941-c04c-4a07-bed6-c012cba8189b\") " pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 16:59:58.903056 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.902981    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx6g9\" (UniqueName: \"kubernetes.io/projected/e0365941-c04c-4a07-bed6-c012cba8189b-kube-api-access-jx6g9\") pod \"success-200-isvc-6a778-predictor-ff6c4454b-ftnlq\" (UID: \"e0365941-c04c-4a07-bed6-c012cba8189b\") " pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 16:59:58.903056 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.903023    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04de9312-8a16-4824-ac77-fad3abef545a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6a778-predictor-84cc69f4d7-sn28q\" (UID: \"04de9312-8a16-4824-ac77-fad3abef545a\") " pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 16:59:58.903474 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.903453    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0365941-c04c-4a07-bed6-c012cba8189b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6a778-predictor-ff6c4454b-ftnlq\" (UID: \"e0365941-c04c-4a07-bed6-c012cba8189b\") " pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 16:59:58.905118 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.905100    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0365941-c04c-4a07-bed6-c012cba8189b-proxy-tls\") pod \"success-200-isvc-6a778-predictor-ff6c4454b-ftnlq\" (UID: \"e0365941-c04c-4a07-bed6-c012cba8189b\") " pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 16:59:58.911210 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.911192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx6g9\" (UniqueName: \"kubernetes.io/projected/e0365941-c04c-4a07-bed6-c012cba8189b-kube-api-access-jx6g9\") pod \"success-200-isvc-6a778-predictor-ff6c4454b-ftnlq\" (UID: \"e0365941-c04c-4a07-bed6-c012cba8189b\") " pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 16:59:58.955586 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:58.955559    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 16:59:59.003774 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.003739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04de9312-8a16-4824-ac77-fad3abef545a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6a778-predictor-84cc69f4d7-sn28q\" (UID: \"04de9312-8a16-4824-ac77-fad3abef545a\") " pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 16:59:59.003911 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.003836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04de9312-8a16-4824-ac77-fad3abef545a-proxy-tls\") pod \"error-404-isvc-6a778-predictor-84cc69f4d7-sn28q\" (UID: \"04de9312-8a16-4824-ac77-fad3abef545a\") " pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 16:59:59.003911 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.003872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzddv\" (UniqueName: \"kubernetes.io/projected/04de9312-8a16-4824-ac77-fad3abef545a-kube-api-access-tzddv\") pod \"error-404-isvc-6a778-predictor-84cc69f4d7-sn28q\" (UID: \"04de9312-8a16-4824-ac77-fad3abef545a\") " pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 16:59:59.004465 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.004445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04de9312-8a16-4824-ac77-fad3abef545a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6a778-predictor-84cc69f4d7-sn28q\" (UID: \"04de9312-8a16-4824-ac77-fad3abef545a\") " pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 16:59:59.006154 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.006135    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04de9312-8a16-4824-ac77-fad3abef545a-proxy-tls\") pod \"error-404-isvc-6a778-predictor-84cc69f4d7-sn28q\" (UID: \"04de9312-8a16-4824-ac77-fad3abef545a\") " pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 16:59:59.011967 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.011949    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzddv\" (UniqueName: \"kubernetes.io/projected/04de9312-8a16-4824-ac77-fad3abef545a-kube-api-access-tzddv\") pod \"error-404-isvc-6a778-predictor-84cc69f4d7-sn28q\" (UID: \"04de9312-8a16-4824-ac77-fad3abef545a\") " pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 16:59:59.044902 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.044837    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 16:59:59.142329 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.142298    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 16:59:59.313953 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.313930    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"]
Sep 15 16:59:59.315861 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:59:59.315836    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0365941_c04c_4a07_bed6_c012cba8189b.slice/crio-1e0a9d39b829148806c9230f43842749fad6854dfb4fffbaf71d58c24c76df03 WatchSource:0}: Error finding container 1e0a9d39b829148806c9230f43842749fad6854dfb4fffbaf71d58c24c76df03: Status 404 returned error can't find the container with id 1e0a9d39b829148806c9230f43842749fad6854dfb4fffbaf71d58c24c76df03
Sep 15 16:59:59.411721 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.411684    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"]
Sep 15 16:59:59.414929 ip-10-0-134-77 kubenswrapper[2565]: W0915 16:59:59.414908    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04de9312_8a16_4824_ac77_fad3abef545a.slice/crio-f35b386e5e574b34fa9847e9127edea722f0b7b2a43bb3a822c850c462735d94 WatchSource:0}: Error finding container f35b386e5e574b34fa9847e9127edea722f0b7b2a43bb3a822c850c462735d94: Status 404 returned error can't find the container with id f35b386e5e574b34fa9847e9127edea722f0b7b2a43bb3a822c850c462735d94
Sep 15 16:59:59.606793 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.606671    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" event={"ID":"e0365941-c04c-4a07-bed6-c012cba8189b","Type":"ContainerStarted","Data":"135838ec3b748ca6bc76044085859ca6672bf7097911d2cb4b9f43861528f543"}
Sep 15 16:59:59.606793 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.606776    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 16:59:59.607259 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.606804    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" event={"ID":"e0365941-c04c-4a07-bed6-c012cba8189b","Type":"ContainerStarted","Data":"b803cd08131903faae5be3fe949699388cae40b9042462543e1741bba6a1049f"}
Sep 15 16:59:59.607259 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.606824    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" event={"ID":"e0365941-c04c-4a07-bed6-c012cba8189b","Type":"ContainerStarted","Data":"1e0a9d39b829148806c9230f43842749fad6854dfb4fffbaf71d58c24c76df03"}
Sep 15 16:59:59.607259 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.606955    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 16:59:59.608527 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.608498    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 16:59:59.608772 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.608749    2565 generic.go:356] "Generic (PLEG): container finished" podID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerID="3292aadd89878676ac294c6a3ce115b33068041a7b4bfd2a8a5b0fc3ec4a26da" exitCode=2
Sep 15 16:59:59.608851 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.608792    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" event={"ID":"caa57fe9-1d64-4cdd-8f12-c02bac09ada0","Type":"ContainerDied","Data":"3292aadd89878676ac294c6a3ce115b33068041a7b4bfd2a8a5b0fc3ec4a26da"}
Sep 15 16:59:59.610750 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.610703    2565 generic.go:356] "Generic (PLEG): container finished" podID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerID="d24622ef4726385a4d54486e0e00d4f2a6fe061d3841808ef5d68268fe956e53" exitCode=2
Sep 15 16:59:59.610887 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.610784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" event={"ID":"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d","Type":"ContainerDied","Data":"d24622ef4726385a4d54486e0e00d4f2a6fe061d3841808ef5d68268fe956e53"}
Sep 15 16:59:59.612569 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.612539    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" event={"ID":"04de9312-8a16-4824-ac77-fad3abef545a","Type":"ContainerStarted","Data":"6300a39abb2e56adbf0f4f4bc6d4a5af93942e90c52ad3bfc3ef87b641e8f076"}
Sep 15 16:59:59.612690 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.612585    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" event={"ID":"04de9312-8a16-4824-ac77-fad3abef545a","Type":"ContainerStarted","Data":"cd99b3d3785650b97c2a5b979430491b9715dc09ecf0a08a397abbf7cccbf165"}
Sep 15 16:59:59.612690 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.612610    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" event={"ID":"04de9312-8a16-4824-ac77-fad3abef545a","Type":"ContainerStarted","Data":"f35b386e5e574b34fa9847e9127edea722f0b7b2a43bb3a822c850c462735d94"}
Sep 15 16:59:59.612690 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.612658    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 16:59:59.612880 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.612697    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 16:59:59.613744 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.613701    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 16:59:59.625444 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.625406    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" podStartSLOduration=1.625393463 podStartE2EDuration="1.625393463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:59:59.624014884 +0000 UTC m=+1990.668400618" watchObservedRunningTime="2026-09-15 16:59:59.625393463 +0000 UTC m=+1990.669779194"
Sep 15 16:59:59.643841 ip-10-0-134-77 kubenswrapper[2565]: I0915 16:59:59.643787    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" podStartSLOduration=1.643773141 podStartE2EDuration="1.643773141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:59:59.641993583 +0000 UTC m=+1990.686379316" watchObservedRunningTime="2026-09-15 16:59:59.643773141 +0000 UTC m=+1990.688158872"
Sep 15 17:00:00.616153 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:00.616108    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 17:00:00.616538 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:00.616229    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 17:00:01.584289 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:01.584244    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 15 17:00:01.589595 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:01.589565    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 17:00:01.936890 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:01.936859    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 17:00:02.024945 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.024916    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwc8p\" (UniqueName: \"kubernetes.io/projected/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-kube-api-access-fwc8p\") pod \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\" (UID: \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\") "
Sep 15 17:00:02.025104 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.025013    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-kube-rbac-proxy-sar-config\") pod \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\" (UID: \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\") "
Sep 15 17:00:02.025104 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.025041    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-proxy-tls\") pod \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\" (UID: \"caa57fe9-1d64-4cdd-8f12-c02bac09ada0\") "
Sep 15 17:00:02.025368 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.025343    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "caa57fe9-1d64-4cdd-8f12-c02bac09ada0" (UID: "caa57fe9-1d64-4cdd-8f12-c02bac09ada0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:00:02.027102 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.027077    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "caa57fe9-1d64-4cdd-8f12-c02bac09ada0" (UID: "caa57fe9-1d64-4cdd-8f12-c02bac09ada0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:00:02.027183 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.027110    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-kube-api-access-fwc8p" (OuterVolumeSpecName: "kube-api-access-fwc8p") pod "caa57fe9-1d64-4cdd-8f12-c02bac09ada0" (UID: "caa57fe9-1d64-4cdd-8f12-c02bac09ada0"). InnerVolumeSpecName "kube-api-access-fwc8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:00:02.126408 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.126321    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:00:02.126408 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.126352    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:00:02.126408 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.126363    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwc8p\" (UniqueName: \"kubernetes.io/projected/caa57fe9-1d64-4cdd-8f12-c02bac09ada0-kube-api-access-fwc8p\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:00:02.404276 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.404252    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 17:00:02.428666 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.428642    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-kube-rbac-proxy-sar-config\") pod \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\" (UID: \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\") "
Sep 15 17:00:02.428830 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.428702    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7vh9\" (UniqueName: \"kubernetes.io/projected/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-kube-api-access-v7vh9\") pod \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\" (UID: \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\") "
Sep 15 17:00:02.428830 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.428764    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-proxy-tls\") pod \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\" (UID: \"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d\") "
Sep 15 17:00:02.429096 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.429068    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" (UID: "898c4f24-2f09-4896-8e8a-7c9ae3f86f6d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:00:02.430754 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.430730    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" (UID: "898c4f24-2f09-4896-8e8a-7c9ae3f86f6d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:00:02.430852 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.430794    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-kube-api-access-v7vh9" (OuterVolumeSpecName: "kube-api-access-v7vh9") pod "898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" (UID: "898c4f24-2f09-4896-8e8a-7c9ae3f86f6d"). InnerVolumeSpecName "kube-api-access-v7vh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:00:02.529484 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.529447    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:00:02.529484 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.529477    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:00:02.529484 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.529487    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7vh9\" (UniqueName: \"kubernetes.io/projected/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d-kube-api-access-v7vh9\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:00:02.624546 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.624513    2565 generic.go:356] "Generic (PLEG): container finished" podID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerID="0eb79e5c06bd5b007b423db32cf134d969fd8012c904942082c407882df841b0" exitCode=0
Sep 15 17:00:02.624751 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.624563    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"
Sep 15 17:00:02.624751 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.624575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" event={"ID":"caa57fe9-1d64-4cdd-8f12-c02bac09ada0","Type":"ContainerDied","Data":"0eb79e5c06bd5b007b423db32cf134d969fd8012c904942082c407882df841b0"}
Sep 15 17:00:02.624751 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.624642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv" event={"ID":"caa57fe9-1d64-4cdd-8f12-c02bac09ada0","Type":"ContainerDied","Data":"73ed7b6148a5e5ea83bdcf3aed24a1a6f0bb3f8de769d8233ce811adf6e39b3d"}
Sep 15 17:00:02.624751 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.624673    2565 scope.go:117] "RemoveContainer" containerID="3292aadd89878676ac294c6a3ce115b33068041a7b4bfd2a8a5b0fc3ec4a26da"
Sep 15 17:00:02.626409 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.626385    2565 generic.go:356] "Generic (PLEG): container finished" podID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerID="88956f7455403d66291da57d410343c3dc8b80dded5293554c10284fa13f50eb" exitCode=0
Sep 15 17:00:02.626520 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.626428    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"
Sep 15 17:00:02.626520 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.626444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" event={"ID":"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d","Type":"ContainerDied","Data":"88956f7455403d66291da57d410343c3dc8b80dded5293554c10284fa13f50eb"}
Sep 15 17:00:02.626520 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.626512    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k" event={"ID":"898c4f24-2f09-4896-8e8a-7c9ae3f86f6d","Type":"ContainerDied","Data":"f2482d4652521330ddd483db0392987e77456f13ac5160979711b6804752308e"}
Sep 15 17:00:02.634176 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.634152    2565 scope.go:117] "RemoveContainer" containerID="0eb79e5c06bd5b007b423db32cf134d969fd8012c904942082c407882df841b0"
Sep 15 17:00:02.643139 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.643117    2565 scope.go:117] "RemoveContainer" containerID="3292aadd89878676ac294c6a3ce115b33068041a7b4bfd2a8a5b0fc3ec4a26da"
Sep 15 17:00:02.643382 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:00:02.643364    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3292aadd89878676ac294c6a3ce115b33068041a7b4bfd2a8a5b0fc3ec4a26da\": container with ID starting with 3292aadd89878676ac294c6a3ce115b33068041a7b4bfd2a8a5b0fc3ec4a26da not found: ID does not exist" containerID="3292aadd89878676ac294c6a3ce115b33068041a7b4bfd2a8a5b0fc3ec4a26da"
Sep 15 17:00:02.643434 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.643390    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3292aadd89878676ac294c6a3ce115b33068041a7b4bfd2a8a5b0fc3ec4a26da"} err="failed to get container status \"3292aadd89878676ac294c6a3ce115b33068041a7b4bfd2a8a5b0fc3ec4a26da\": rpc error: code = NotFound desc = could not find container \"3292aadd89878676ac294c6a3ce115b33068041a7b4bfd2a8a5b0fc3ec4a26da\": container with ID starting with 3292aadd89878676ac294c6a3ce115b33068041a7b4bfd2a8a5b0fc3ec4a26da not found: ID does not exist"
Sep 15 17:00:02.643434 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.643405    2565 scope.go:117] "RemoveContainer" containerID="0eb79e5c06bd5b007b423db32cf134d969fd8012c904942082c407882df841b0"
Sep 15 17:00:02.643675 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:00:02.643657    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb79e5c06bd5b007b423db32cf134d969fd8012c904942082c407882df841b0\": container with ID starting with 0eb79e5c06bd5b007b423db32cf134d969fd8012c904942082c407882df841b0 not found: ID does not exist" containerID="0eb79e5c06bd5b007b423db32cf134d969fd8012c904942082c407882df841b0"
Sep 15 17:00:02.643810 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.643681    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb79e5c06bd5b007b423db32cf134d969fd8012c904942082c407882df841b0"} err="failed to get container status \"0eb79e5c06bd5b007b423db32cf134d969fd8012c904942082c407882df841b0\": rpc error: code = NotFound desc = could not find container \"0eb79e5c06bd5b007b423db32cf134d969fd8012c904942082c407882df841b0\": container with ID starting with 0eb79e5c06bd5b007b423db32cf134d969fd8012c904942082c407882df841b0 not found: ID does not exist"
Sep 15 17:00:02.643810 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.643696    2565 scope.go:117] "RemoveContainer" containerID="d24622ef4726385a4d54486e0e00d4f2a6fe061d3841808ef5d68268fe956e53"
Sep 15 17:00:02.647080 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.647028    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"]
Sep 15 17:00:02.649472 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.649452    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0591a-predictor-dcb65fbbf-8qnfv"]
Sep 15 17:00:02.651158 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.651133    2565 scope.go:117] "RemoveContainer" containerID="88956f7455403d66291da57d410343c3dc8b80dded5293554c10284fa13f50eb"
Sep 15 17:00:02.658038 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.658023    2565 scope.go:117] "RemoveContainer" containerID="d24622ef4726385a4d54486e0e00d4f2a6fe061d3841808ef5d68268fe956e53"
Sep 15 17:00:02.658270 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:00:02.658249    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24622ef4726385a4d54486e0e00d4f2a6fe061d3841808ef5d68268fe956e53\": container with ID starting with d24622ef4726385a4d54486e0e00d4f2a6fe061d3841808ef5d68268fe956e53 not found: ID does not exist" containerID="d24622ef4726385a4d54486e0e00d4f2a6fe061d3841808ef5d68268fe956e53"
Sep 15 17:00:02.658355 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.658276    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24622ef4726385a4d54486e0e00d4f2a6fe061d3841808ef5d68268fe956e53"} err="failed to get container status \"d24622ef4726385a4d54486e0e00d4f2a6fe061d3841808ef5d68268fe956e53\": rpc error: code = NotFound desc = could not find container \"d24622ef4726385a4d54486e0e00d4f2a6fe061d3841808ef5d68268fe956e53\": container with ID starting with d24622ef4726385a4d54486e0e00d4f2a6fe061d3841808ef5d68268fe956e53 not found: ID does not exist"
Sep 15 17:00:02.658355 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.658290    2565 scope.go:117] "RemoveContainer" containerID="88956f7455403d66291da57d410343c3dc8b80dded5293554c10284fa13f50eb"
Sep 15 17:00:02.658569 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:00:02.658545    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88956f7455403d66291da57d410343c3dc8b80dded5293554c10284fa13f50eb\": container with ID starting with 88956f7455403d66291da57d410343c3dc8b80dded5293554c10284fa13f50eb not found: ID does not exist" containerID="88956f7455403d66291da57d410343c3dc8b80dded5293554c10284fa13f50eb"
Sep 15 17:00:02.658653 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.658575    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88956f7455403d66291da57d410343c3dc8b80dded5293554c10284fa13f50eb"} err="failed to get container status \"88956f7455403d66291da57d410343c3dc8b80dded5293554c10284fa13f50eb\": rpc error: code = NotFound desc = could not find container \"88956f7455403d66291da57d410343c3dc8b80dded5293554c10284fa13f50eb\": container with ID starting with 88956f7455403d66291da57d410343c3dc8b80dded5293554c10284fa13f50eb not found: ID does not exist"
Sep 15 17:00:02.660578 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.660539    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"]
Sep 15 17:00:02.661978 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:02.661956    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0591a-predictor-78848d555b-bpb6k"]
Sep 15 17:00:03.449365 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:03.449332    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" path="/var/lib/kubelet/pods/898c4f24-2f09-4896-8e8a-7c9ae3f86f6d/volumes"
Sep 15 17:00:03.449777 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:03.449763    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" path="/var/lib/kubelet/pods/caa57fe9-1d64-4cdd-8f12-c02bac09ada0/volumes"
Sep 15 17:00:05.620691 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:05.620663    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 17:00:05.621190 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:05.621157    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 17:00:05.621257 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:05.621181    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 17:00:05.621575 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:05.621550    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 17:00:15.621642 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:15.621596    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 17:00:15.622050 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:15.621596    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 17:00:25.621516 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:25.621478    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 17:00:25.621916 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:25.621608    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 17:00:35.621185 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:35.621143    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 17:00:35.621677 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:35.621575    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 17:00:38.561373 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.561338    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:00:38.655126 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.655096    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"]
Sep 15 17:00:38.655389 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.655360    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kserve-container" containerID="cri-o://449532a53b02c756d34d9d11af26e0cb3fe1f559d084f67713c17f9dea6f40f2" gracePeriod=30
Sep 15 17:00:38.655481 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.655399    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kube-rbac-proxy" containerID="cri-o://819320e70a2a52b76f5ff712c471cc77167a6d14dc39723b42455b4a28934e75" gracePeriod=30
Sep 15 17:00:38.686554 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.686522    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"]
Sep 15 17:00:38.686825 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.686780    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kserve-container" containerID="cri-o://73f7552e5a6483062937b2ec8b272b9ef0fcee6764fc0da6f25ec0b96eaf00ca" gracePeriod=30
Sep 15 17:00:38.686943 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.686889    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kube-rbac-proxy" containerID="cri-o://556638e337a0c99d31bc5074da1e4ad457758594f3aa5cb37464e64a74e8570f" gracePeriod=30
Sep 15 17:00:38.690655 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.690613    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:00:38.783363 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.783326    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:00:38.785436 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.785410    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"]
Sep 15 17:00:38.785837 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.785817    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kserve-container"
Sep 15 17:00:38.785921 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.785839    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kserve-container"
Sep 15 17:00:38.785921 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.785848    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kserve-container"
Sep 15 17:00:38.785921 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.785853    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kserve-container"
Sep 15 17:00:38.785921 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.785862    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kube-rbac-proxy"
Sep 15 17:00:38.785921 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.785867    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kube-rbac-proxy"
Sep 15 17:00:38.785921 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.785878    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kube-rbac-proxy"
Sep 15 17:00:38.785921 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.785883    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kube-rbac-proxy"
Sep 15 17:00:38.786149 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.785947    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kube-rbac-proxy"
Sep 15 17:00:38.786149 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.785955    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kube-rbac-proxy"
Sep 15 17:00:38.786149 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.785964    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="caa57fe9-1d64-4cdd-8f12-c02bac09ada0" containerName="kserve-container"
Sep 15 17:00:38.786149 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.785969    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="898c4f24-2f09-4896-8e8a-7c9ae3f86f6d" containerName="kserve-container"
Sep 15 17:00:38.788917 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.788903    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:38.791663 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.791639    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-969ee-predictor-serving-cert\""
Sep 15 17:00:38.791789 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.791645    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-969ee-kube-rbac-proxy-sar-config\""
Sep 15 17:00:38.798725 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.798685    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"]
Sep 15 17:00:38.867537 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.867452    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:00:38.879824 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.879799    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"]
Sep 15 17:00:38.883113 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.883097    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:38.885904 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.885835    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-969ee-predictor-serving-cert\""
Sep 15 17:00:38.885904 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.885860    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-969ee-kube-rbac-proxy-sar-config\""
Sep 15 17:00:38.892822 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.892801    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"]
Sep 15 17:00:38.905499 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.905473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lznfk\" (UniqueName: \"kubernetes.io/projected/615f9876-5359-4d83-82cd-56a11fafe4fc-kube-api-access-lznfk\") pod \"success-200-isvc-969ee-predictor-84b4498f4d-sfpj9\" (UID: \"615f9876-5359-4d83-82cd-56a11fafe4fc\") " pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:38.905596 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.905522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/615f9876-5359-4d83-82cd-56a11fafe4fc-proxy-tls\") pod \"success-200-isvc-969ee-predictor-84b4498f4d-sfpj9\" (UID: \"615f9876-5359-4d83-82cd-56a11fafe4fc\") " pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:38.905652 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.905614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/615f9876-5359-4d83-82cd-56a11fafe4fc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-969ee-predictor-84b4498f4d-sfpj9\" (UID: \"615f9876-5359-4d83-82cd-56a11fafe4fc\") " pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:38.937152 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:38.937105    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:00:39.006552 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.006514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lznfk\" (UniqueName: \"kubernetes.io/projected/615f9876-5359-4d83-82cd-56a11fafe4fc-kube-api-access-lznfk\") pod \"success-200-isvc-969ee-predictor-84b4498f4d-sfpj9\" (UID: \"615f9876-5359-4d83-82cd-56a11fafe4fc\") " pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:39.006702 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.006573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/615f9876-5359-4d83-82cd-56a11fafe4fc-proxy-tls\") pod \"success-200-isvc-969ee-predictor-84b4498f4d-sfpj9\" (UID: \"615f9876-5359-4d83-82cd-56a11fafe4fc\") " pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:39.006702 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.006612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddea253e-be1c-4d28-a8ea-b1c52990edec-proxy-tls\") pod \"error-404-isvc-969ee-predictor-7bffb9f7df-jdmch\" (UID: \"ddea253e-be1c-4d28-a8ea-b1c52990edec\") " pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:39.006702 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.006637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jkf2\" (UniqueName: \"kubernetes.io/projected/ddea253e-be1c-4d28-a8ea-b1c52990edec-kube-api-access-9jkf2\") pod \"error-404-isvc-969ee-predictor-7bffb9f7df-jdmch\" (UID: \"ddea253e-be1c-4d28-a8ea-b1c52990edec\") " pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:39.007195 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.006775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddea253e-be1c-4d28-a8ea-b1c52990edec-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-969ee-predictor-7bffb9f7df-jdmch\" (UID: \"ddea253e-be1c-4d28-a8ea-b1c52990edec\") " pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:39.007195 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:00:39.006784    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-969ee-predictor-serving-cert: secret "success-200-isvc-969ee-predictor-serving-cert" not found
Sep 15 17:00:39.007195 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.006836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/615f9876-5359-4d83-82cd-56a11fafe4fc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-969ee-predictor-84b4498f4d-sfpj9\" (UID: \"615f9876-5359-4d83-82cd-56a11fafe4fc\") " pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:39.007195 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:00:39.006853    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/615f9876-5359-4d83-82cd-56a11fafe4fc-proxy-tls podName:615f9876-5359-4d83-82cd-56a11fafe4fc nodeName:}" failed. No retries permitted until 2026-09-15 17:00:39.506830316 +0000 UTC m=+2030.551216027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/615f9876-5359-4d83-82cd-56a11fafe4fc-proxy-tls") pod "success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" (UID: "615f9876-5359-4d83-82cd-56a11fafe4fc") : secret "success-200-isvc-969ee-predictor-serving-cert" not found
Sep 15 17:00:39.007516 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.007494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/615f9876-5359-4d83-82cd-56a11fafe4fc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-969ee-predictor-84b4498f4d-sfpj9\" (UID: \"615f9876-5359-4d83-82cd-56a11fafe4fc\") " pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:39.015523 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.015489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lznfk\" (UniqueName: \"kubernetes.io/projected/615f9876-5359-4d83-82cd-56a11fafe4fc-kube-api-access-lznfk\") pod \"success-200-isvc-969ee-predictor-84b4498f4d-sfpj9\" (UID: \"615f9876-5359-4d83-82cd-56a11fafe4fc\") " pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:39.107314 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.107280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddea253e-be1c-4d28-a8ea-b1c52990edec-proxy-tls\") pod \"error-404-isvc-969ee-predictor-7bffb9f7df-jdmch\" (UID: \"ddea253e-be1c-4d28-a8ea-b1c52990edec\") " pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:39.107314 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.107313    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jkf2\" (UniqueName: \"kubernetes.io/projected/ddea253e-be1c-4d28-a8ea-b1c52990edec-kube-api-access-9jkf2\") pod \"error-404-isvc-969ee-predictor-7bffb9f7df-jdmch\" (UID: \"ddea253e-be1c-4d28-a8ea-b1c52990edec\") " pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:39.107563 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.107334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddea253e-be1c-4d28-a8ea-b1c52990edec-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-969ee-predictor-7bffb9f7df-jdmch\" (UID: \"ddea253e-be1c-4d28-a8ea-b1c52990edec\") " pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:39.107563 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:00:39.107452    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-969ee-predictor-serving-cert: secret "error-404-isvc-969ee-predictor-serving-cert" not found
Sep 15 17:00:39.107563 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:00:39.107522    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddea253e-be1c-4d28-a8ea-b1c52990edec-proxy-tls podName:ddea253e-be1c-4d28-a8ea-b1c52990edec nodeName:}" failed. No retries permitted until 2026-09-15 17:00:39.607503446 +0000 UTC m=+2030.651889171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ddea253e-be1c-4d28-a8ea-b1c52990edec-proxy-tls") pod "error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" (UID: "ddea253e-be1c-4d28-a8ea-b1c52990edec") : secret "error-404-isvc-969ee-predictor-serving-cert" not found
Sep 15 17:00:39.108077 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.108058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddea253e-be1c-4d28-a8ea-b1c52990edec-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-969ee-predictor-7bffb9f7df-jdmch\" (UID: \"ddea253e-be1c-4d28-a8ea-b1c52990edec\") " pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:39.116367 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.116336    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jkf2\" (UniqueName: \"kubernetes.io/projected/ddea253e-be1c-4d28-a8ea-b1c52990edec-kube-api-access-9jkf2\") pod \"error-404-isvc-969ee-predictor-7bffb9f7df-jdmch\" (UID: \"ddea253e-be1c-4d28-a8ea-b1c52990edec\") " pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:39.175412 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.175379    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:00:39.510987 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.510892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/615f9876-5359-4d83-82cd-56a11fafe4fc-proxy-tls\") pod \"success-200-isvc-969ee-predictor-84b4498f4d-sfpj9\" (UID: \"615f9876-5359-4d83-82cd-56a11fafe4fc\") " pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:39.513352 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.513326    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/615f9876-5359-4d83-82cd-56a11fafe4fc-proxy-tls\") pod \"success-200-isvc-969ee-predictor-84b4498f4d-sfpj9\" (UID: \"615f9876-5359-4d83-82cd-56a11fafe4fc\") " pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:39.542623 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.542593    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:00:39.611766 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.611725    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddea253e-be1c-4d28-a8ea-b1c52990edec-proxy-tls\") pod \"error-404-isvc-969ee-predictor-7bffb9f7df-jdmch\" (UID: \"ddea253e-be1c-4d28-a8ea-b1c52990edec\") " pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:39.614146 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.614126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddea253e-be1c-4d28-a8ea-b1c52990edec-proxy-tls\") pod \"error-404-isvc-969ee-predictor-7bffb9f7df-jdmch\" (UID: \"ddea253e-be1c-4d28-a8ea-b1c52990edec\") " pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:39.701793 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.701752    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:39.772043 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.771958    2565 generic.go:356] "Generic (PLEG): container finished" podID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerID="819320e70a2a52b76f5ff712c471cc77167a6d14dc39723b42455b4a28934e75" exitCode=2
Sep 15 17:00:39.772043 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.771995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" event={"ID":"7564c558-9f3b-4f34-b01f-5051282ddc26","Type":"ContainerDied","Data":"819320e70a2a52b76f5ff712c471cc77167a6d14dc39723b42455b4a28934e75"}
Sep 15 17:00:39.774453 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.774427    2565 generic.go:356] "Generic (PLEG): container finished" podID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerID="556638e337a0c99d31bc5074da1e4ad457758594f3aa5cb37464e64a74e8570f" exitCode=2
Sep 15 17:00:39.774566 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.774486    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" event={"ID":"8323660f-7c26-4c6a-a210-0cf03e067abe","Type":"ContainerDied","Data":"556638e337a0c99d31bc5074da1e4ad457758594f3aa5cb37464e64a74e8570f"}
Sep 15 17:00:39.793846 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.793824    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:39.973790 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:39.973760    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"]
Sep 15 17:00:39.976582 ip-10-0-134-77 kubenswrapper[2565]: W0915 17:00:39.976549    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615f9876_5359_4d83_82cd_56a11fafe4fc.slice/crio-0d2c95da3b116a70a7f89d40aba34827a924402694c7fdfd9768f074c333e8b9 WatchSource:0}: Error finding container 0d2c95da3b116a70a7f89d40aba34827a924402694c7fdfd9768f074c333e8b9: Status 404 returned error can't find the container with id 0d2c95da3b116a70a7f89d40aba34827a924402694c7fdfd9768f074c333e8b9
Sep 15 17:00:40.070492 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.070385    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"]
Sep 15 17:00:40.080535 ip-10-0-134-77 kubenswrapper[2565]: W0915 17:00:40.080510    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddea253e_be1c_4d28_a8ea_b1c52990edec.slice/crio-167066463ac0d49d6b8c6a15031244b783e02c187805d00833a3784e472db977 WatchSource:0}: Error finding container 167066463ac0d49d6b8c6a15031244b783e02c187805d00833a3784e472db977: Status 404 returned error can't find the container with id 167066463ac0d49d6b8c6a15031244b783e02c187805d00833a3784e472db977
Sep 15 17:00:40.779693 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.779646    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" event={"ID":"ddea253e-be1c-4d28-a8ea-b1c52990edec","Type":"ContainerStarted","Data":"21ef353105fb4a69034665db254ec4ad5ed64531826bc8c70851aa10ed6beb5e"}
Sep 15 17:00:40.780185 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.779728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" event={"ID":"ddea253e-be1c-4d28-a8ea-b1c52990edec","Type":"ContainerStarted","Data":"3fd9860cceb588519d5e2b739b09684927fdea8448959d1e248ce7464231a576"}
Sep 15 17:00:40.780185 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.779759    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" event={"ID":"ddea253e-be1c-4d28-a8ea-b1c52990edec","Type":"ContainerStarted","Data":"167066463ac0d49d6b8c6a15031244b783e02c187805d00833a3784e472db977"}
Sep 15 17:00:40.780185 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.779791    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:40.780185 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.779821    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:40.780668 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.780638    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 15 17:00:40.781415 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.781388    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 17:00:40.781733 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.781696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" event={"ID":"615f9876-5359-4d83-82cd-56a11fafe4fc","Type":"ContainerStarted","Data":"f6c18b456dbaf760e3827f0fc0eebea4e8832b193ad091ae18f5c35dcd7254e9"}
Sep 15 17:00:40.781796 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.781766    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" event={"ID":"615f9876-5359-4d83-82cd-56a11fafe4fc","Type":"ContainerStarted","Data":"259447bda86c0478ce97f1fbdb5338fa061f7c8adae6026b1781f42b73f7302e"}
Sep 15 17:00:40.781796 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.781789    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" event={"ID":"615f9876-5359-4d83-82cd-56a11fafe4fc","Type":"ContainerStarted","Data":"0d2c95da3b116a70a7f89d40aba34827a924402694c7fdfd9768f074c333e8b9"}
Sep 15 17:00:40.781879 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.781815    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:40.786344 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.786317    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 17:00:40.796231 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.796181    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" podStartSLOduration=2.796166165 podStartE2EDuration="2.796166165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:00:40.795748135 +0000 UTC m=+2031.840133871" watchObservedRunningTime="2026-09-15 17:00:40.796166165 +0000 UTC m=+2031.840551895"
Sep 15 17:00:40.812849 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:40.812808    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" podStartSLOduration=2.8127943699999998 podStartE2EDuration="2.81279437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:00:40.811213316 +0000 UTC m=+2031.855599048" watchObservedRunningTime="2026-09-15 17:00:40.81279437 +0000 UTC m=+2031.857180102"
Sep 15 17:00:41.785250 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:41.785195    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 15 17:00:41.785749 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:41.785298    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 17:00:41.785749 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:41.785419    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:41.786684 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:41.786659    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 17:00:41.790505 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:41.790480    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 17:00:42.030012 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.029988    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 17:00:42.132959 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.132852    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f97sj\" (UniqueName: \"kubernetes.io/projected/8323660f-7c26-4c6a-a210-0cf03e067abe-kube-api-access-f97sj\") pod \"8323660f-7c26-4c6a-a210-0cf03e067abe\" (UID: \"8323660f-7c26-4c6a-a210-0cf03e067abe\") "
Sep 15 17:00:42.132959 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.132900    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8323660f-7c26-4c6a-a210-0cf03e067abe-proxy-tls\") pod \"8323660f-7c26-4c6a-a210-0cf03e067abe\" (UID: \"8323660f-7c26-4c6a-a210-0cf03e067abe\") "
Sep 15 17:00:42.132959 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.132951    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8323660f-7c26-4c6a-a210-0cf03e067abe-kube-rbac-proxy-sar-config\") pod \"8323660f-7c26-4c6a-a210-0cf03e067abe\" (UID: \"8323660f-7c26-4c6a-a210-0cf03e067abe\") "
Sep 15 17:00:42.133472 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.133442    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8323660f-7c26-4c6a-a210-0cf03e067abe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8323660f-7c26-4c6a-a210-0cf03e067abe" (UID: "8323660f-7c26-4c6a-a210-0cf03e067abe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:00:42.135564 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.135527    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8323660f-7c26-4c6a-a210-0cf03e067abe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8323660f-7c26-4c6a-a210-0cf03e067abe" (UID: "8323660f-7c26-4c6a-a210-0cf03e067abe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:00:42.135754 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.135666    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8323660f-7c26-4c6a-a210-0cf03e067abe-kube-api-access-f97sj" (OuterVolumeSpecName: "kube-api-access-f97sj") pod "8323660f-7c26-4c6a-a210-0cf03e067abe" (UID: "8323660f-7c26-4c6a-a210-0cf03e067abe"). InnerVolumeSpecName "kube-api-access-f97sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:00:42.234414 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.234245    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8323660f-7c26-4c6a-a210-0cf03e067abe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:00:42.234414 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.234283    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f97sj\" (UniqueName: \"kubernetes.io/projected/8323660f-7c26-4c6a-a210-0cf03e067abe-kube-api-access-f97sj\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:00:42.234414 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.234301    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8323660f-7c26-4c6a-a210-0cf03e067abe-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:00:42.290766 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.290742    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 17:00:42.436812 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.436699    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7564c558-9f3b-4f34-b01f-5051282ddc26-proxy-tls\") pod \"7564c558-9f3b-4f34-b01f-5051282ddc26\" (UID: \"7564c558-9f3b-4f34-b01f-5051282ddc26\") "
Sep 15 17:00:42.436812 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.436800    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdxbn\" (UniqueName: \"kubernetes.io/projected/7564c558-9f3b-4f34-b01f-5051282ddc26-kube-api-access-qdxbn\") pod \"7564c558-9f3b-4f34-b01f-5051282ddc26\" (UID: \"7564c558-9f3b-4f34-b01f-5051282ddc26\") "
Sep 15 17:00:42.437120 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.436856    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7564c558-9f3b-4f34-b01f-5051282ddc26-kube-rbac-proxy-sar-config\") pod \"7564c558-9f3b-4f34-b01f-5051282ddc26\" (UID: \"7564c558-9f3b-4f34-b01f-5051282ddc26\") "
Sep 15 17:00:42.437291 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.437238    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7564c558-9f3b-4f34-b01f-5051282ddc26-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7564c558-9f3b-4f34-b01f-5051282ddc26" (UID: "7564c558-9f3b-4f34-b01f-5051282ddc26"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:00:42.437441 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.437420    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7564c558-9f3b-4f34-b01f-5051282ddc26-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:00:42.439369 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.439343    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7564c558-9f3b-4f34-b01f-5051282ddc26-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7564c558-9f3b-4f34-b01f-5051282ddc26" (UID: "7564c558-9f3b-4f34-b01f-5051282ddc26"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:00:42.439523 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.439498    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7564c558-9f3b-4f34-b01f-5051282ddc26-kube-api-access-qdxbn" (OuterVolumeSpecName: "kube-api-access-qdxbn") pod "7564c558-9f3b-4f34-b01f-5051282ddc26" (UID: "7564c558-9f3b-4f34-b01f-5051282ddc26"). InnerVolumeSpecName "kube-api-access-qdxbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:00:42.538810 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.538771    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdxbn\" (UniqueName: \"kubernetes.io/projected/7564c558-9f3b-4f34-b01f-5051282ddc26-kube-api-access-qdxbn\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:00:42.538810 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.538807    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7564c558-9f3b-4f34-b01f-5051282ddc26-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:00:42.790224 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.790134    2565 generic.go:356] "Generic (PLEG): container finished" podID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerID="449532a53b02c756d34d9d11af26e0cb3fe1f559d084f67713c17f9dea6f40f2" exitCode=0
Sep 15 17:00:42.790224 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.790190    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"
Sep 15 17:00:42.790699 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.790227    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" event={"ID":"7564c558-9f3b-4f34-b01f-5051282ddc26","Type":"ContainerDied","Data":"449532a53b02c756d34d9d11af26e0cb3fe1f559d084f67713c17f9dea6f40f2"}
Sep 15 17:00:42.790699 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.790294    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v" event={"ID":"7564c558-9f3b-4f34-b01f-5051282ddc26","Type":"ContainerDied","Data":"37b344ec8c2de253050e240d08f1891613f8b1f8be88c0e64e361034edf3d6c2"}
Sep 15 17:00:42.790699 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.790336    2565 scope.go:117] "RemoveContainer" containerID="819320e70a2a52b76f5ff712c471cc77167a6d14dc39723b42455b4a28934e75"
Sep 15 17:00:42.792048 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.792026    2565 generic.go:356] "Generic (PLEG): container finished" podID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerID="73f7552e5a6483062937b2ec8b272b9ef0fcee6764fc0da6f25ec0b96eaf00ca" exitCode=0
Sep 15 17:00:42.792164 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.792076    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" event={"ID":"8323660f-7c26-4c6a-a210-0cf03e067abe","Type":"ContainerDied","Data":"73f7552e5a6483062937b2ec8b272b9ef0fcee6764fc0da6f25ec0b96eaf00ca"}
Sep 15 17:00:42.792164 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.792088    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"
Sep 15 17:00:42.792164 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.792124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9" event={"ID":"8323660f-7c26-4c6a-a210-0cf03e067abe","Type":"ContainerDied","Data":"886c9b628c5476bcb9b0e723ee68e62a29feaf28d58e012bf0e6f2860ebe211e"}
Sep 15 17:00:42.792424 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.792405    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 17:00:42.798543 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.798519    2565 scope.go:117] "RemoveContainer" containerID="449532a53b02c756d34d9d11af26e0cb3fe1f559d084f67713c17f9dea6f40f2"
Sep 15 17:00:42.805981 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.805962    2565 scope.go:117] "RemoveContainer" containerID="819320e70a2a52b76f5ff712c471cc77167a6d14dc39723b42455b4a28934e75"
Sep 15 17:00:42.806259 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:00:42.806241    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819320e70a2a52b76f5ff712c471cc77167a6d14dc39723b42455b4a28934e75\": container with ID starting with 819320e70a2a52b76f5ff712c471cc77167a6d14dc39723b42455b4a28934e75 not found: ID does not exist" containerID="819320e70a2a52b76f5ff712c471cc77167a6d14dc39723b42455b4a28934e75"
Sep 15 17:00:42.806316 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.806268    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819320e70a2a52b76f5ff712c471cc77167a6d14dc39723b42455b4a28934e75"} err="failed to get container status \"819320e70a2a52b76f5ff712c471cc77167a6d14dc39723b42455b4a28934e75\": rpc error: code = NotFound desc = could not find container \"819320e70a2a52b76f5ff712c471cc77167a6d14dc39723b42455b4a28934e75\": container with ID starting with 819320e70a2a52b76f5ff712c471cc77167a6d14dc39723b42455b4a28934e75 not found: ID does not exist"
Sep 15 17:00:42.806316 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.806283    2565 scope.go:117] "RemoveContainer" containerID="449532a53b02c756d34d9d11af26e0cb3fe1f559d084f67713c17f9dea6f40f2"
Sep 15 17:00:42.806539 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:00:42.806521    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449532a53b02c756d34d9d11af26e0cb3fe1f559d084f67713c17f9dea6f40f2\": container with ID starting with 449532a53b02c756d34d9d11af26e0cb3fe1f559d084f67713c17f9dea6f40f2 not found: ID does not exist" containerID="449532a53b02c756d34d9d11af26e0cb3fe1f559d084f67713c17f9dea6f40f2"
Sep 15 17:00:42.806584 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.806546    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449532a53b02c756d34d9d11af26e0cb3fe1f559d084f67713c17f9dea6f40f2"} err="failed to get container status \"449532a53b02c756d34d9d11af26e0cb3fe1f559d084f67713c17f9dea6f40f2\": rpc error: code = NotFound desc = could not find container \"449532a53b02c756d34d9d11af26e0cb3fe1f559d084f67713c17f9dea6f40f2\": container with ID starting with 449532a53b02c756d34d9d11af26e0cb3fe1f559d084f67713c17f9dea6f40f2 not found: ID does not exist"
Sep 15 17:00:42.806584 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.806562    2565 scope.go:117] "RemoveContainer" containerID="556638e337a0c99d31bc5074da1e4ad457758594f3aa5cb37464e64a74e8570f"
Sep 15 17:00:42.813546 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.813526    2565 scope.go:117] "RemoveContainer" containerID="73f7552e5a6483062937b2ec8b272b9ef0fcee6764fc0da6f25ec0b96eaf00ca"
Sep 15 17:00:42.814612 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.814590    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"]
Sep 15 17:00:42.818508 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.818487    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d406-predictor-9dcd77c94-qkb6v"]
Sep 15 17:00:42.822452 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.822435    2565 scope.go:117] "RemoveContainer" containerID="556638e337a0c99d31bc5074da1e4ad457758594f3aa5cb37464e64a74e8570f"
Sep 15 17:00:42.822797 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:00:42.822698    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556638e337a0c99d31bc5074da1e4ad457758594f3aa5cb37464e64a74e8570f\": container with ID starting with 556638e337a0c99d31bc5074da1e4ad457758594f3aa5cb37464e64a74e8570f not found: ID does not exist" containerID="556638e337a0c99d31bc5074da1e4ad457758594f3aa5cb37464e64a74e8570f"
Sep 15 17:00:42.822857 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.822806    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556638e337a0c99d31bc5074da1e4ad457758594f3aa5cb37464e64a74e8570f"} err="failed to get container status \"556638e337a0c99d31bc5074da1e4ad457758594f3aa5cb37464e64a74e8570f\": rpc error: code = NotFound desc = could not find container \"556638e337a0c99d31bc5074da1e4ad457758594f3aa5cb37464e64a74e8570f\": container with ID starting with 556638e337a0c99d31bc5074da1e4ad457758594f3aa5cb37464e64a74e8570f not found: ID does not exist"
Sep 15 17:00:42.822857 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.822823    2565 scope.go:117] "RemoveContainer" containerID="73f7552e5a6483062937b2ec8b272b9ef0fcee6764fc0da6f25ec0b96eaf00ca"
Sep 15 17:00:42.823085 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:00:42.823068    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f7552e5a6483062937b2ec8b272b9ef0fcee6764fc0da6f25ec0b96eaf00ca\": container with ID starting with 73f7552e5a6483062937b2ec8b272b9ef0fcee6764fc0da6f25ec0b96eaf00ca not found: ID does not exist" containerID="73f7552e5a6483062937b2ec8b272b9ef0fcee6764fc0da6f25ec0b96eaf00ca"
Sep 15 17:00:42.823125 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.823090    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f7552e5a6483062937b2ec8b272b9ef0fcee6764fc0da6f25ec0b96eaf00ca"} err="failed to get container status \"73f7552e5a6483062937b2ec8b272b9ef0fcee6764fc0da6f25ec0b96eaf00ca\": rpc error: code = NotFound desc = could not find container \"73f7552e5a6483062937b2ec8b272b9ef0fcee6764fc0da6f25ec0b96eaf00ca\": container with ID starting with 73f7552e5a6483062937b2ec8b272b9ef0fcee6764fc0da6f25ec0b96eaf00ca not found: ID does not exist"
Sep 15 17:00:42.830516 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.830496    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"]
Sep 15 17:00:42.834231 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:42.834209    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d406-predictor-c94cb4b67-9qvp9"]
Sep 15 17:00:43.449287 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:43.449256    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" path="/var/lib/kubelet/pods/7564c558-9f3b-4f34-b01f-5051282ddc26/volumes"
Sep 15 17:00:43.449663 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:43.449651    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" path="/var/lib/kubelet/pods/8323660f-7c26-4c6a-a210-0cf03e067abe/volumes"
Sep 15 17:00:45.621912 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:45.621871    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 17:00:45.622619 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:45.622600    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 17:00:46.790416 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:46.790388    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:00:46.790912 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:46.790886    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 17:00:47.797306 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:47.797268    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:00:47.797841 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:47.797813    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 17:00:56.791786 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:56.791742    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 17:00:57.798766 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:57.798703    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 17:00:58.796409 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:58.796368    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:00:58.955815 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:00:58.955775    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:01:06.791798 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:06.791758    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 17:01:07.798388 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:07.798340    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 17:01:08.843326 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:08.843239    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:01:08.910093 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:08.910064    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"]
Sep 15 17:01:08.910316 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:08.910277    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kserve-container" containerID="cri-o://b803cd08131903faae5be3fe949699388cae40b9042462543e1741bba6a1049f" gracePeriod=30
Sep 15 17:01:08.910389 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:08.910301    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kube-rbac-proxy" containerID="cri-o://135838ec3b748ca6bc76044085859ca6672bf7097911d2cb4b9f43861528f543" gracePeriod=30
Sep 15 17:01:08.938448 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:08.938395    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:01:08.938609 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:08.938476    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"]
Sep 15 17:01:08.938909 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:08.938879    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kserve-container" containerID="cri-o://cd99b3d3785650b97c2a5b979430491b9715dc09ecf0a08a397abbf7cccbf165" gracePeriod=30
Sep 15 17:01:08.939027 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:08.938913    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kube-rbac-proxy" containerID="cri-o://6300a39abb2e56adbf0f4f4bc6d4a5af93942e90c52ad3bfc3ef87b641e8f076" gracePeriod=30
Sep 15 17:01:09.004278 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.004247    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"]
Sep 15 17:01:09.004540 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.004529    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kube-rbac-proxy"
Sep 15 17:01:09.004586 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.004542    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kube-rbac-proxy"
Sep 15 17:01:09.004586 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.004552    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kserve-container"
Sep 15 17:01:09.004586 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.004559    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kserve-container"
Sep 15 17:01:09.004586 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.004567    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kserve-container"
Sep 15 17:01:09.004586 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.004572    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kserve-container"
Sep 15 17:01:09.004586 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.004581    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kube-rbac-proxy"
Sep 15 17:01:09.004586 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.004586    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kube-rbac-proxy"
Sep 15 17:01:09.004832 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.004632    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kserve-container"
Sep 15 17:01:09.004832 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.004641    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7564c558-9f3b-4f34-b01f-5051282ddc26" containerName="kube-rbac-proxy"
Sep 15 17:01:09.004832 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.004647    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kserve-container"
Sep 15 17:01:09.004832 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.004652    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8323660f-7c26-4c6a-a210-0cf03e067abe" containerName="kube-rbac-proxy"
Sep 15 17:01:09.006998 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.006976    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:01:09.007948 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.007933    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:09.010624 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.010606    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1f299-predictor-serving-cert\""
Sep 15 17:01:09.010736 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.010720    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1f299-kube-rbac-proxy-sar-config\""
Sep 15 17:01:09.018472 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.018451    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"]
Sep 15 17:01:09.018957 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.018932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b429c763-f770-4b6e-a198-966171a160e0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj\" (UID: \"b429c763-f770-4b6e-a198-966171a160e0\") " pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:09.019048 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.018981    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b429c763-f770-4b6e-a198-966171a160e0-proxy-tls\") pod \"success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj\" (UID: \"b429c763-f770-4b6e-a198-966171a160e0\") " pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:09.019048 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.019011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xmbs\" (UniqueName: \"kubernetes.io/projected/b429c763-f770-4b6e-a198-966171a160e0-kube-api-access-7xmbs\") pod \"success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj\" (UID: \"b429c763-f770-4b6e-a198-966171a160e0\") " pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:09.086999 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.086958    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:01:09.093927 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.093868    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"]
Sep 15 17:01:09.097652 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.097630    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:09.100257 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.100233    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1f299-kube-rbac-proxy-sar-config\""
Sep 15 17:01:09.100398 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.100239    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1f299-predictor-serving-cert\""
Sep 15 17:01:09.105241 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.105219    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"]
Sep 15 17:01:09.120001 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.119971    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b429c763-f770-4b6e-a198-966171a160e0-proxy-tls\") pod \"success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj\" (UID: \"b429c763-f770-4b6e-a198-966171a160e0\") " pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:09.120118 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.120011    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xmbs\" (UniqueName: \"kubernetes.io/projected/b429c763-f770-4b6e-a198-966171a160e0-kube-api-access-7xmbs\") pod \"success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj\" (UID: \"b429c763-f770-4b6e-a198-966171a160e0\") " pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:09.120118 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.120084    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76n84\" (UniqueName: \"kubernetes.io/projected/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-kube-api-access-76n84\") pod \"error-404-isvc-1f299-predictor-576964885f-fklcx\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") " pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:09.120118 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:01:09.120109    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-1f299-predictor-serving-cert: secret "success-200-isvc-1f299-predictor-serving-cert" not found
Sep 15 17:01:09.120277 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.120120    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1f299-predictor-576964885f-fklcx\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") " pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:09.120277 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:01:09.120168    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b429c763-f770-4b6e-a198-966171a160e0-proxy-tls podName:b429c763-f770-4b6e-a198-966171a160e0 nodeName:}" failed. No retries permitted until 2026-09-15 17:01:09.620150134 +0000 UTC m=+2060.664535860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b429c763-f770-4b6e-a198-966171a160e0-proxy-tls") pod "success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" (UID: "b429c763-f770-4b6e-a198-966171a160e0") : secret "success-200-isvc-1f299-predictor-serving-cert" not found
Sep 15 17:01:09.120277 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.120252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-proxy-tls\") pod \"error-404-isvc-1f299-predictor-576964885f-fklcx\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") " pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:09.120430 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.120301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b429c763-f770-4b6e-a198-966171a160e0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj\" (UID: \"b429c763-f770-4b6e-a198-966171a160e0\") " pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:09.120883 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.120866    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b429c763-f770-4b6e-a198-966171a160e0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj\" (UID: \"b429c763-f770-4b6e-a198-966171a160e0\") " pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:09.128878 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.128859    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xmbs\" (UniqueName: \"kubernetes.io/projected/b429c763-f770-4b6e-a198-966171a160e0-kube-api-access-7xmbs\") pod \"success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj\" (UID: \"b429c763-f770-4b6e-a198-966171a160e0\") " pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:09.220696 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.220661    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-proxy-tls\") pod \"error-404-isvc-1f299-predictor-576964885f-fklcx\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") " pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:09.220937 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.220798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76n84\" (UniqueName: \"kubernetes.io/projected/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-kube-api-access-76n84\") pod \"error-404-isvc-1f299-predictor-576964885f-fklcx\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") " pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:09.220937 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.220838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1f299-predictor-576964885f-fklcx\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") " pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:09.220937 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:01:09.220841    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1f299-predictor-serving-cert: secret "error-404-isvc-1f299-predictor-serving-cert" not found
Sep 15 17:01:09.220937 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:01:09.220919    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-proxy-tls podName:8727bf0f-7379-4192-a1f2-c5f1d6ef9e91 nodeName:}" failed. No retries permitted until 2026-09-15 17:01:09.720898164 +0000 UTC m=+2060.765283881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-proxy-tls") pod "error-404-isvc-1f299-predictor-576964885f-fklcx" (UID: "8727bf0f-7379-4192-a1f2-c5f1d6ef9e91") : secret "error-404-isvc-1f299-predictor-serving-cert" not found
Sep 15 17:01:09.221563 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.221539    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1f299-predictor-576964885f-fklcx\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") " pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:09.232419 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.232371    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76n84\" (UniqueName: \"kubernetes.io/projected/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-kube-api-access-76n84\") pod \"error-404-isvc-1f299-predictor-576964885f-fklcx\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") " pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:09.609904 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.609870    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:01:09.625258 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.625231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b429c763-f770-4b6e-a198-966171a160e0-proxy-tls\") pod \"success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj\" (UID: \"b429c763-f770-4b6e-a198-966171a160e0\") " pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:09.627851 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.627829    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b429c763-f770-4b6e-a198-966171a160e0-proxy-tls\") pod \"success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj\" (UID: \"b429c763-f770-4b6e-a198-966171a160e0\") " pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:09.726226 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.726193    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-proxy-tls\") pod \"error-404-isvc-1f299-predictor-576964885f-fklcx\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") " pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:09.726422 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:01:09.726322    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1f299-predictor-serving-cert: secret "error-404-isvc-1f299-predictor-serving-cert" not found
Sep 15 17:01:09.726422 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:01:09.726385    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-proxy-tls podName:8727bf0f-7379-4192-a1f2-c5f1d6ef9e91 nodeName:}" failed. No retries permitted until 2026-09-15 17:01:10.726368575 +0000 UTC m=+2061.770754285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-proxy-tls") pod "error-404-isvc-1f299-predictor-576964885f-fklcx" (UID: "8727bf0f-7379-4192-a1f2-c5f1d6ef9e91") : secret "error-404-isvc-1f299-predictor-serving-cert" not found
Sep 15 17:01:09.896199 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.896163    2565 generic.go:356] "Generic (PLEG): container finished" podID="04de9312-8a16-4824-ac77-fad3abef545a" containerID="6300a39abb2e56adbf0f4f4bc6d4a5af93942e90c52ad3bfc3ef87b641e8f076" exitCode=2
Sep 15 17:01:09.896619 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.896241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" event={"ID":"04de9312-8a16-4824-ac77-fad3abef545a","Type":"ContainerDied","Data":"6300a39abb2e56adbf0f4f4bc6d4a5af93942e90c52ad3bfc3ef87b641e8f076"}
Sep 15 17:01:09.897969 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.897945    2565 generic.go:356] "Generic (PLEG): container finished" podID="e0365941-c04c-4a07-bed6-c012cba8189b" containerID="135838ec3b748ca6bc76044085859ca6672bf7097911d2cb4b9f43861528f543" exitCode=2
Sep 15 17:01:09.898095 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.897998    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" event={"ID":"e0365941-c04c-4a07-bed6-c012cba8189b","Type":"ContainerDied","Data":"135838ec3b748ca6bc76044085859ca6672bf7097911d2cb4b9f43861528f543"}
Sep 15 17:01:09.919255 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:09.919228    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:10.028005 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.027973    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:01:10.191686 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.191662    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"]
Sep 15 17:01:10.194052 ip-10-0-134-77 kubenswrapper[2565]: W0915 17:01:10.194021    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb429c763_f770_4b6e_a198_966171a160e0.slice/crio-8a318334cba5e39c9c867977f35c3617797fb5c760666eca286cb9b3219c7762 WatchSource:0}: Error finding container 8a318334cba5e39c9c867977f35c3617797fb5c760666eca286cb9b3219c7762: Status 404 returned error can't find the container with id 8a318334cba5e39c9c867977f35c3617797fb5c760666eca286cb9b3219c7762
Sep 15 17:01:10.617134 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.617023    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 15 17:01:10.617134 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.617023    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 15 17:01:10.734196 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.734156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-proxy-tls\") pod \"error-404-isvc-1f299-predictor-576964885f-fklcx\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") " pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:10.736594 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.736566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-proxy-tls\") pod \"error-404-isvc-1f299-predictor-576964885f-fklcx\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") " pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:10.903945 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.903912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" event={"ID":"b429c763-f770-4b6e-a198-966171a160e0","Type":"ContainerStarted","Data":"fe78e18a00c316b5ff7aa2d740cebbf5a3673831754e8d6af89cfccc2bf8fa7a"}
Sep 15 17:01:10.904319 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.904006    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:10.904319 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.904030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" event={"ID":"b429c763-f770-4b6e-a198-966171a160e0","Type":"ContainerStarted","Data":"4973120cb1aee2f2a173d049f11f9523fc042088af2e234c0a67211f7c8795b1"}
Sep 15 17:01:10.904319 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.904049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" event={"ID":"b429c763-f770-4b6e-a198-966171a160e0","Type":"ContainerStarted","Data":"8a318334cba5e39c9c867977f35c3617797fb5c760666eca286cb9b3219c7762"}
Sep 15 17:01:10.904319 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.904078    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:10.905316 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.905291    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 17:01:10.910956 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.910939    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:10.921866 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:10.921817    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" podStartSLOduration=2.92180055 podStartE2EDuration="2.92180055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:01:10.921158195 +0000 UTC m=+2061.965543928" watchObservedRunningTime="2026-09-15 17:01:10.92180055 +0000 UTC m=+2061.966186282"
Sep 15 17:01:11.181631 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:11.181550    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"]
Sep 15 17:01:11.184499 ip-10-0-134-77 kubenswrapper[2565]: W0915 17:01:11.184468    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8727bf0f_7379_4192_a1f2_c5f1d6ef9e91.slice/crio-b076ad955b57cb6366d387e72d191bb3e89974ba4f56f7033316cdc369a63788 WatchSource:0}: Error finding container b076ad955b57cb6366d387e72d191bb3e89974ba4f56f7033316cdc369a63788: Status 404 returned error can't find the container with id b076ad955b57cb6366d387e72d191bb3e89974ba4f56f7033316cdc369a63788
Sep 15 17:01:11.909389 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:11.909353    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" event={"ID":"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91","Type":"ContainerStarted","Data":"f2e53bf95a1ba77c4daaf77a34052f8a59773a28f02393a64fc5c19be04e07ac"}
Sep 15 17:01:11.909806 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:11.909418    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" event={"ID":"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91","Type":"ContainerStarted","Data":"24ca898b748effc69b213cd36762eaeab6702e75fa4c56b8fb7a47f646355464"}
Sep 15 17:01:11.909806 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:11.909441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" event={"ID":"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91","Type":"ContainerStarted","Data":"b076ad955b57cb6366d387e72d191bb3e89974ba4f56f7033316cdc369a63788"}
Sep 15 17:01:11.909806 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:11.909574    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:11.909806 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:11.909626    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 17:01:11.910028 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:11.909808    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:11.910976 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:11.910956    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 17:01:11.926547 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:11.926501    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" podStartSLOduration=2.926488648 podStartE2EDuration="2.926488648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:01:11.925989764 +0000 UTC m=+2062.970375509" watchObservedRunningTime="2026-09-15 17:01:11.926488648 +0000 UTC m=+2062.970874380"
Sep 15 17:01:12.741979 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.741948    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 17:01:12.751856 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.751765    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04de9312-8a16-4824-ac77-fad3abef545a-kube-rbac-proxy-sar-config\") pod \"04de9312-8a16-4824-ac77-fad3abef545a\" (UID: \"04de9312-8a16-4824-ac77-fad3abef545a\") "
Sep 15 17:01:12.752008 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.751881    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzddv\" (UniqueName: \"kubernetes.io/projected/04de9312-8a16-4824-ac77-fad3abef545a-kube-api-access-tzddv\") pod \"04de9312-8a16-4824-ac77-fad3abef545a\" (UID: \"04de9312-8a16-4824-ac77-fad3abef545a\") "
Sep 15 17:01:12.752008 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.751930    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04de9312-8a16-4824-ac77-fad3abef545a-proxy-tls\") pod \"04de9312-8a16-4824-ac77-fad3abef545a\" (UID: \"04de9312-8a16-4824-ac77-fad3abef545a\") "
Sep 15 17:01:12.752432 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.752388    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04de9312-8a16-4824-ac77-fad3abef545a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "04de9312-8a16-4824-ac77-fad3abef545a" (UID: "04de9312-8a16-4824-ac77-fad3abef545a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:01:12.754538 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.754515    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04de9312-8a16-4824-ac77-fad3abef545a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04de9312-8a16-4824-ac77-fad3abef545a" (UID: "04de9312-8a16-4824-ac77-fad3abef545a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:01:12.754632 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.754537    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04de9312-8a16-4824-ac77-fad3abef545a-kube-api-access-tzddv" (OuterVolumeSpecName: "kube-api-access-tzddv") pod "04de9312-8a16-4824-ac77-fad3abef545a" (UID: "04de9312-8a16-4824-ac77-fad3abef545a"). InnerVolumeSpecName "kube-api-access-tzddv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:01:12.852859 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.852829    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzddv\" (UniqueName: \"kubernetes.io/projected/04de9312-8a16-4824-ac77-fad3abef545a-kube-api-access-tzddv\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:01:12.852859 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.852856    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04de9312-8a16-4824-ac77-fad3abef545a-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:01:12.853075 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.852867    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04de9312-8a16-4824-ac77-fad3abef545a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:01:12.857032 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.857008    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 17:01:12.917660 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.917629    2565 generic.go:356] "Generic (PLEG): container finished" podID="04de9312-8a16-4824-ac77-fad3abef545a" containerID="cd99b3d3785650b97c2a5b979430491b9715dc09ecf0a08a397abbf7cccbf165" exitCode=0
Sep 15 17:01:12.918068 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.917675    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"
Sep 15 17:01:12.918068 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.917737    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" event={"ID":"04de9312-8a16-4824-ac77-fad3abef545a","Type":"ContainerDied","Data":"cd99b3d3785650b97c2a5b979430491b9715dc09ecf0a08a397abbf7cccbf165"}
Sep 15 17:01:12.918068 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.917792    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q" event={"ID":"04de9312-8a16-4824-ac77-fad3abef545a","Type":"ContainerDied","Data":"f35b386e5e574b34fa9847e9127edea722f0b7b2a43bb3a822c850c462735d94"}
Sep 15 17:01:12.918068 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.917823    2565 scope.go:117] "RemoveContainer" containerID="6300a39abb2e56adbf0f4f4bc6d4a5af93942e90c52ad3bfc3ef87b641e8f076"
Sep 15 17:01:12.919625 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.919601    2565 generic.go:356] "Generic (PLEG): container finished" podID="e0365941-c04c-4a07-bed6-c012cba8189b" containerID="b803cd08131903faae5be3fe949699388cae40b9042462543e1741bba6a1049f" exitCode=0
Sep 15 17:01:12.919751 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.919641    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"
Sep 15 17:01:12.919751 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.919672    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" event={"ID":"e0365941-c04c-4a07-bed6-c012cba8189b","Type":"ContainerDied","Data":"b803cd08131903faae5be3fe949699388cae40b9042462543e1741bba6a1049f"}
Sep 15 17:01:12.919751 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.919738    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq" event={"ID":"e0365941-c04c-4a07-bed6-c012cba8189b","Type":"ContainerDied","Data":"1e0a9d39b829148806c9230f43842749fad6854dfb4fffbaf71d58c24c76df03"}
Sep 15 17:01:12.920166 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.920142    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 17:01:12.926222 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.926205    2565 scope.go:117] "RemoveContainer" containerID="cd99b3d3785650b97c2a5b979430491b9715dc09ecf0a08a397abbf7cccbf165"
Sep 15 17:01:12.934409 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.934369    2565 scope.go:117] "RemoveContainer" containerID="6300a39abb2e56adbf0f4f4bc6d4a5af93942e90c52ad3bfc3ef87b641e8f076"
Sep 15 17:01:12.934738 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:01:12.934698    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6300a39abb2e56adbf0f4f4bc6d4a5af93942e90c52ad3bfc3ef87b641e8f076\": container with ID starting with 6300a39abb2e56adbf0f4f4bc6d4a5af93942e90c52ad3bfc3ef87b641e8f076 not found: ID does not exist" containerID="6300a39abb2e56adbf0f4f4bc6d4a5af93942e90c52ad3bfc3ef87b641e8f076"
Sep 15 17:01:12.934850 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.934746    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6300a39abb2e56adbf0f4f4bc6d4a5af93942e90c52ad3bfc3ef87b641e8f076"} err="failed to get container status \"6300a39abb2e56adbf0f4f4bc6d4a5af93942e90c52ad3bfc3ef87b641e8f076\": rpc error: code = NotFound desc = could not find container \"6300a39abb2e56adbf0f4f4bc6d4a5af93942e90c52ad3bfc3ef87b641e8f076\": container with ID starting with 6300a39abb2e56adbf0f4f4bc6d4a5af93942e90c52ad3bfc3ef87b641e8f076 not found: ID does not exist"
Sep 15 17:01:12.934850 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.934767    2565 scope.go:117] "RemoveContainer" containerID="cd99b3d3785650b97c2a5b979430491b9715dc09ecf0a08a397abbf7cccbf165"
Sep 15 17:01:12.935083 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:01:12.935064    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd99b3d3785650b97c2a5b979430491b9715dc09ecf0a08a397abbf7cccbf165\": container with ID starting with cd99b3d3785650b97c2a5b979430491b9715dc09ecf0a08a397abbf7cccbf165 not found: ID does not exist" containerID="cd99b3d3785650b97c2a5b979430491b9715dc09ecf0a08a397abbf7cccbf165"
Sep 15 17:01:12.935155 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.935089    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd99b3d3785650b97c2a5b979430491b9715dc09ecf0a08a397abbf7cccbf165"} err="failed to get container status \"cd99b3d3785650b97c2a5b979430491b9715dc09ecf0a08a397abbf7cccbf165\": rpc error: code = NotFound desc = could not find container \"cd99b3d3785650b97c2a5b979430491b9715dc09ecf0a08a397abbf7cccbf165\": container with ID starting with cd99b3d3785650b97c2a5b979430491b9715dc09ecf0a08a397abbf7cccbf165 not found: ID does not exist"
Sep 15 17:01:12.935155 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.935108    2565 scope.go:117] "RemoveContainer" containerID="135838ec3b748ca6bc76044085859ca6672bf7097911d2cb4b9f43861528f543"
Sep 15 17:01:12.940935 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.940909    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"]
Sep 15 17:01:12.943658 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.943643    2565 scope.go:117] "RemoveContainer" containerID="b803cd08131903faae5be3fe949699388cae40b9042462543e1741bba6a1049f"
Sep 15 17:01:12.944780 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.944755    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6a778-predictor-84cc69f4d7-sn28q"]
Sep 15 17:01:12.950673 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.950648    2565 scope.go:117] "RemoveContainer" containerID="135838ec3b748ca6bc76044085859ca6672bf7097911d2cb4b9f43861528f543"
Sep 15 17:01:12.950979 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:01:12.950962    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135838ec3b748ca6bc76044085859ca6672bf7097911d2cb4b9f43861528f543\": container with ID starting with 135838ec3b748ca6bc76044085859ca6672bf7097911d2cb4b9f43861528f543 not found: ID does not exist" containerID="135838ec3b748ca6bc76044085859ca6672bf7097911d2cb4b9f43861528f543"
Sep 15 17:01:12.951054 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.950986    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135838ec3b748ca6bc76044085859ca6672bf7097911d2cb4b9f43861528f543"} err="failed to get container status \"135838ec3b748ca6bc76044085859ca6672bf7097911d2cb4b9f43861528f543\": rpc error: code = NotFound desc = could not find container \"135838ec3b748ca6bc76044085859ca6672bf7097911d2cb4b9f43861528f543\": container with ID starting with 135838ec3b748ca6bc76044085859ca6672bf7097911d2cb4b9f43861528f543 not found: ID does not exist"
Sep 15 17:01:12.951054 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.951001    2565 scope.go:117] "RemoveContainer" containerID="b803cd08131903faae5be3fe949699388cae40b9042462543e1741bba6a1049f"
Sep 15 17:01:12.951228 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:01:12.951213    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b803cd08131903faae5be3fe949699388cae40b9042462543e1741bba6a1049f\": container with ID starting with b803cd08131903faae5be3fe949699388cae40b9042462543e1741bba6a1049f not found: ID does not exist" containerID="b803cd08131903faae5be3fe949699388cae40b9042462543e1741bba6a1049f"
Sep 15 17:01:12.951268 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.951232    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b803cd08131903faae5be3fe949699388cae40b9042462543e1741bba6a1049f"} err="failed to get container status \"b803cd08131903faae5be3fe949699388cae40b9042462543e1741bba6a1049f\": rpc error: code = NotFound desc = could not find container \"b803cd08131903faae5be3fe949699388cae40b9042462543e1741bba6a1049f\": container with ID starting with b803cd08131903faae5be3fe949699388cae40b9042462543e1741bba6a1049f not found: ID does not exist"
Sep 15 17:01:12.953482 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.953464    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx6g9\" (UniqueName: \"kubernetes.io/projected/e0365941-c04c-4a07-bed6-c012cba8189b-kube-api-access-jx6g9\") pod \"e0365941-c04c-4a07-bed6-c012cba8189b\" (UID: \"e0365941-c04c-4a07-bed6-c012cba8189b\") "
Sep 15 17:01:12.953547 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.953536    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0365941-c04c-4a07-bed6-c012cba8189b-kube-rbac-proxy-sar-config\") pod \"e0365941-c04c-4a07-bed6-c012cba8189b\" (UID: \"e0365941-c04c-4a07-bed6-c012cba8189b\") "
Sep 15 17:01:12.953586 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.953564    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0365941-c04c-4a07-bed6-c012cba8189b-proxy-tls\") pod \"e0365941-c04c-4a07-bed6-c012cba8189b\" (UID: \"e0365941-c04c-4a07-bed6-c012cba8189b\") "
Sep 15 17:01:12.953942 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.953915    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0365941-c04c-4a07-bed6-c012cba8189b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0365941-c04c-4a07-bed6-c012cba8189b" (UID: "e0365941-c04c-4a07-bed6-c012cba8189b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:01:12.955420 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.955394    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0365941-c04c-4a07-bed6-c012cba8189b-kube-api-access-jx6g9" (OuterVolumeSpecName: "kube-api-access-jx6g9") pod "e0365941-c04c-4a07-bed6-c012cba8189b" (UID: "e0365941-c04c-4a07-bed6-c012cba8189b"). InnerVolumeSpecName "kube-api-access-jx6g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:01:12.955496 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:12.955442    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0365941-c04c-4a07-bed6-c012cba8189b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0365941-c04c-4a07-bed6-c012cba8189b" (UID: "e0365941-c04c-4a07-bed6-c012cba8189b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:01:13.054982 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:13.054945    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0365941-c04c-4a07-bed6-c012cba8189b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:01:13.054982 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:13.054975    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0365941-c04c-4a07-bed6-c012cba8189b-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:01:13.054982 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:13.054987    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx6g9\" (UniqueName: \"kubernetes.io/projected/e0365941-c04c-4a07-bed6-c012cba8189b-kube-api-access-jx6g9\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:01:13.241722 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:13.241678    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"]
Sep 15 17:01:13.245078 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:13.245058    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6a778-predictor-ff6c4454b-ftnlq"]
Sep 15 17:01:13.448846 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:13.448811    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04de9312-8a16-4824-ac77-fad3abef545a" path="/var/lib/kubelet/pods/04de9312-8a16-4824-ac77-fad3abef545a/volumes"
Sep 15 17:01:13.449255 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:13.449242    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" path="/var/lib/kubelet/pods/e0365941-c04c-4a07-bed6-c012cba8189b/volumes"
Sep 15 17:01:13.925723 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:13.925678    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 17:01:16.790878 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:16.790835    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 17:01:16.914838 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:16.914807    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:01:16.915351 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:16.915327    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 17:01:17.798410 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:17.798373    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 17:01:18.930246 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:18.930215    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:01:18.930895 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:18.930850    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 17:01:26.791792 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:26.791759    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:01:26.915534 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:26.915499    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 17:01:27.798868 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:27.798837    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:01:28.931628 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:28.931578    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 17:01:36.916183 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:36.916141    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 17:01:38.791122 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:38.791093    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:01:38.931262 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:38.931218    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 17:01:39.273155 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:39.273110    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:01:46.915759 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:46.915702    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 17:01:48.931146 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:48.931102    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 17:01:56.915466 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:56.915416    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 17:01:58.931873 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:01:58.931840    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:02:06.916696 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:02:06.916661    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:02:19.077469 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:02:19.077411    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:02:19.177094 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:02:19.177056    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:04:08.802832 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:04:08.802702    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:06:08.886242 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:06:08.886195    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:06:48.511621 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:06:48.511589    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:10:33.823608 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:33.823568    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:10:33.885511 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:33.885480    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"]
Sep 15 17:10:33.885740 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:33.885700    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kserve-container" containerID="cri-o://4973120cb1aee2f2a173d049f11f9523fc042088af2e234c0a67211f7c8795b1" gracePeriod=30
Sep 15 17:10:33.885820 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:33.885751    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kube-rbac-proxy" containerID="cri-o://fe78e18a00c316b5ff7aa2d740cebbf5a3673831754e8d6af89cfccc2bf8fa7a" gracePeriod=30
Sep 15 17:10:33.918150 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:33.918122    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"]
Sep 15 17:10:33.918415 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:33.918381    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kserve-container" containerID="cri-o://24ca898b748effc69b213cd36762eaeab6702e75fa4c56b8fb7a47f646355464" gracePeriod=30
Sep 15 17:10:33.918561 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:33.918458    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kube-rbac-proxy" containerID="cri-o://f2e53bf95a1ba77c4daaf77a34052f8a59773a28f02393a64fc5c19be04e07ac" gracePeriod=30
Sep 15 17:10:33.926333 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:33.926306    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 15 17:10:33.935922 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:33.935895    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:10:34.034496 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:34.034467    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:10:34.940732 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:34.940673    2565 generic.go:356] "Generic (PLEG): container finished" podID="b429c763-f770-4b6e-a198-966171a160e0" containerID="fe78e18a00c316b5ff7aa2d740cebbf5a3673831754e8d6af89cfccc2bf8fa7a" exitCode=2
Sep 15 17:10:34.941170 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:34.940754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" event={"ID":"b429c763-f770-4b6e-a198-966171a160e0","Type":"ContainerDied","Data":"fe78e18a00c316b5ff7aa2d740cebbf5a3673831754e8d6af89cfccc2bf8fa7a"}
Sep 15 17:10:34.942654 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:34.942626    2565 generic.go:356] "Generic (PLEG): container finished" podID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerID="f2e53bf95a1ba77c4daaf77a34052f8a59773a28f02393a64fc5c19be04e07ac" exitCode=2
Sep 15 17:10:34.942800 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:34.942689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" event={"ID":"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91","Type":"ContainerDied","Data":"f2e53bf95a1ba77c4daaf77a34052f8a59773a28f02393a64fc5c19be04e07ac"}
Sep 15 17:10:36.824955 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:36.824932    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:10:36.951915 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:36.951883    2565 generic.go:356] "Generic (PLEG): container finished" podID="b429c763-f770-4b6e-a198-966171a160e0" containerID="4973120cb1aee2f2a173d049f11f9523fc042088af2e234c0a67211f7c8795b1" exitCode=0
Sep 15 17:10:36.952024 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:36.951963    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"
Sep 15 17:10:36.952072 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:36.951962    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" event={"ID":"b429c763-f770-4b6e-a198-966171a160e0","Type":"ContainerDied","Data":"4973120cb1aee2f2a173d049f11f9523fc042088af2e234c0a67211f7c8795b1"}
Sep 15 17:10:36.952203 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:36.952120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj" event={"ID":"b429c763-f770-4b6e-a198-966171a160e0","Type":"ContainerDied","Data":"8a318334cba5e39c9c867977f35c3617797fb5c760666eca286cb9b3219c7762"}
Sep 15 17:10:36.952267 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:36.952227    2565 scope.go:117] "RemoveContainer" containerID="fe78e18a00c316b5ff7aa2d740cebbf5a3673831754e8d6af89cfccc2bf8fa7a"
Sep 15 17:10:36.970171 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:36.970148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b429c763-f770-4b6e-a198-966171a160e0-kube-rbac-proxy-sar-config\") pod \"b429c763-f770-4b6e-a198-966171a160e0\" (UID: \"b429c763-f770-4b6e-a198-966171a160e0\") "
Sep 15 17:10:36.970264 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:36.970179    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b429c763-f770-4b6e-a198-966171a160e0-proxy-tls\") pod \"b429c763-f770-4b6e-a198-966171a160e0\" (UID: \"b429c763-f770-4b6e-a198-966171a160e0\") "
Sep 15 17:10:36.970264 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:36.970221    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xmbs\" (UniqueName: \"kubernetes.io/projected/b429c763-f770-4b6e-a198-966171a160e0-kube-api-access-7xmbs\") pod \"b429c763-f770-4b6e-a198-966171a160e0\" (UID: \"b429c763-f770-4b6e-a198-966171a160e0\") "
Sep 15 17:10:36.970507 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:36.970482    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b429c763-f770-4b6e-a198-966171a160e0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b429c763-f770-4b6e-a198-966171a160e0" (UID: "b429c763-f770-4b6e-a198-966171a160e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:10:36.972232 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:36.972213    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b429c763-f770-4b6e-a198-966171a160e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b429c763-f770-4b6e-a198-966171a160e0" (UID: "b429c763-f770-4b6e-a198-966171a160e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:10:36.972348 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:36.972332    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b429c763-f770-4b6e-a198-966171a160e0-kube-api-access-7xmbs" (OuterVolumeSpecName: "kube-api-access-7xmbs") pod "b429c763-f770-4b6e-a198-966171a160e0" (UID: "b429c763-f770-4b6e-a198-966171a160e0"). InnerVolumeSpecName "kube-api-access-7xmbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:10:37.028089 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.028062    2565 scope.go:117] "RemoveContainer" containerID="4973120cb1aee2f2a173d049f11f9523fc042088af2e234c0a67211f7c8795b1"
Sep 15 17:10:37.035116 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.035100    2565 scope.go:117] "RemoveContainer" containerID="fe78e18a00c316b5ff7aa2d740cebbf5a3673831754e8d6af89cfccc2bf8fa7a"
Sep 15 17:10:37.035468 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:10:37.035448    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe78e18a00c316b5ff7aa2d740cebbf5a3673831754e8d6af89cfccc2bf8fa7a\": container with ID starting with fe78e18a00c316b5ff7aa2d740cebbf5a3673831754e8d6af89cfccc2bf8fa7a not found: ID does not exist" containerID="fe78e18a00c316b5ff7aa2d740cebbf5a3673831754e8d6af89cfccc2bf8fa7a"
Sep 15 17:10:37.035537 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.035477    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe78e18a00c316b5ff7aa2d740cebbf5a3673831754e8d6af89cfccc2bf8fa7a"} err="failed to get container status \"fe78e18a00c316b5ff7aa2d740cebbf5a3673831754e8d6af89cfccc2bf8fa7a\": rpc error: code = NotFound desc = could not find container \"fe78e18a00c316b5ff7aa2d740cebbf5a3673831754e8d6af89cfccc2bf8fa7a\": container with ID starting with fe78e18a00c316b5ff7aa2d740cebbf5a3673831754e8d6af89cfccc2bf8fa7a not found: ID does not exist"
Sep 15 17:10:37.035537 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.035495    2565 scope.go:117] "RemoveContainer" containerID="4973120cb1aee2f2a173d049f11f9523fc042088af2e234c0a67211f7c8795b1"
Sep 15 17:10:37.035800 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:10:37.035778    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4973120cb1aee2f2a173d049f11f9523fc042088af2e234c0a67211f7c8795b1\": container with ID starting with 4973120cb1aee2f2a173d049f11f9523fc042088af2e234c0a67211f7c8795b1 not found: ID does not exist" containerID="4973120cb1aee2f2a173d049f11f9523fc042088af2e234c0a67211f7c8795b1"
Sep 15 17:10:37.035847 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.035811    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4973120cb1aee2f2a173d049f11f9523fc042088af2e234c0a67211f7c8795b1"} err="failed to get container status \"4973120cb1aee2f2a173d049f11f9523fc042088af2e234c0a67211f7c8795b1\": rpc error: code = NotFound desc = could not find container \"4973120cb1aee2f2a173d049f11f9523fc042088af2e234c0a67211f7c8795b1\": container with ID starting with 4973120cb1aee2f2a173d049f11f9523fc042088af2e234c0a67211f7c8795b1 not found: ID does not exist"
Sep 15 17:10:37.071365 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.071325    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b429c763-f770-4b6e-a198-966171a160e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:10:37.071365 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.071367    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b429c763-f770-4b6e-a198-966171a160e0-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:10:37.071548 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.071385    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xmbs\" (UniqueName: \"kubernetes.io/projected/b429c763-f770-4b6e-a198-966171a160e0-kube-api-access-7xmbs\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:10:37.150564 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.150540    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:10:37.272215 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.272186    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-proxy-tls\") pod \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") "
Sep 15 17:10:37.272369 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.272235    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76n84\" (UniqueName: \"kubernetes.io/projected/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-kube-api-access-76n84\") pod \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") "
Sep 15 17:10:37.272369 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.272263    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-kube-rbac-proxy-sar-config\") pod \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\" (UID: \"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91\") "
Sep 15 17:10:37.272532 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.272509    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"]
Sep 15 17:10:37.272650 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.272617    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" (UID: "8727bf0f-7379-4192-a1f2-c5f1d6ef9e91"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:10:37.274364 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.274342    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-kube-api-access-76n84" (OuterVolumeSpecName: "kube-api-access-76n84") pod "8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" (UID: "8727bf0f-7379-4192-a1f2-c5f1d6ef9e91"). InnerVolumeSpecName "kube-api-access-76n84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:10:37.274454 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.274432    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" (UID: "8727bf0f-7379-4192-a1f2-c5f1d6ef9e91"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:10:37.276377 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.276356    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1f299-predictor-5f5c5bc97d-gfzpj"]
Sep 15 17:10:37.373660 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.373626    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:10:37.373660 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.373652    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76n84\" (UniqueName: \"kubernetes.io/projected/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-kube-api-access-76n84\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:10:37.373660 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.373661    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:10:37.448641 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.448608    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b429c763-f770-4b6e-a198-966171a160e0" path="/var/lib/kubelet/pods/b429c763-f770-4b6e-a198-966171a160e0/volumes"
Sep 15 17:10:37.957234 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.957151    2565 generic.go:356] "Generic (PLEG): container finished" podID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerID="24ca898b748effc69b213cd36762eaeab6702e75fa4c56b8fb7a47f646355464" exitCode=0
Sep 15 17:10:37.957234 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.957201    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"
Sep 15 17:10:37.957743 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.957236    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" event={"ID":"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91","Type":"ContainerDied","Data":"24ca898b748effc69b213cd36762eaeab6702e75fa4c56b8fb7a47f646355464"}
Sep 15 17:10:37.957743 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.957298    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx" event={"ID":"8727bf0f-7379-4192-a1f2-c5f1d6ef9e91","Type":"ContainerDied","Data":"b076ad955b57cb6366d387e72d191bb3e89974ba4f56f7033316cdc369a63788"}
Sep 15 17:10:37.957743 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.957336    2565 scope.go:117] "RemoveContainer" containerID="f2e53bf95a1ba77c4daaf77a34052f8a59773a28f02393a64fc5c19be04e07ac"
Sep 15 17:10:37.964960 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.964942    2565 scope.go:117] "RemoveContainer" containerID="24ca898b748effc69b213cd36762eaeab6702e75fa4c56b8fb7a47f646355464"
Sep 15 17:10:37.971814 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.971793    2565 scope.go:117] "RemoveContainer" containerID="f2e53bf95a1ba77c4daaf77a34052f8a59773a28f02393a64fc5c19be04e07ac"
Sep 15 17:10:37.972095 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:10:37.972075    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e53bf95a1ba77c4daaf77a34052f8a59773a28f02393a64fc5c19be04e07ac\": container with ID starting with f2e53bf95a1ba77c4daaf77a34052f8a59773a28f02393a64fc5c19be04e07ac not found: ID does not exist" containerID="f2e53bf95a1ba77c4daaf77a34052f8a59773a28f02393a64fc5c19be04e07ac"
Sep 15 17:10:37.972142 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.972106    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e53bf95a1ba77c4daaf77a34052f8a59773a28f02393a64fc5c19be04e07ac"} err="failed to get container status \"f2e53bf95a1ba77c4daaf77a34052f8a59773a28f02393a64fc5c19be04e07ac\": rpc error: code = NotFound desc = could not find container \"f2e53bf95a1ba77c4daaf77a34052f8a59773a28f02393a64fc5c19be04e07ac\": container with ID starting with f2e53bf95a1ba77c4daaf77a34052f8a59773a28f02393a64fc5c19be04e07ac not found: ID does not exist"
Sep 15 17:10:37.972142 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.972126    2565 scope.go:117] "RemoveContainer" containerID="24ca898b748effc69b213cd36762eaeab6702e75fa4c56b8fb7a47f646355464"
Sep 15 17:10:37.972366 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:10:37.972352    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ca898b748effc69b213cd36762eaeab6702e75fa4c56b8fb7a47f646355464\": container with ID starting with 24ca898b748effc69b213cd36762eaeab6702e75fa4c56b8fb7a47f646355464 not found: ID does not exist" containerID="24ca898b748effc69b213cd36762eaeab6702e75fa4c56b8fb7a47f646355464"
Sep 15 17:10:37.972409 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.972371    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ca898b748effc69b213cd36762eaeab6702e75fa4c56b8fb7a47f646355464"} err="failed to get container status \"24ca898b748effc69b213cd36762eaeab6702e75fa4c56b8fb7a47f646355464\": rpc error: code = NotFound desc = could not find container \"24ca898b748effc69b213cd36762eaeab6702e75fa4c56b8fb7a47f646355464\": container with ID starting with 24ca898b748effc69b213cd36762eaeab6702e75fa4c56b8fb7a47f646355464 not found: ID does not exist"
Sep 15 17:10:37.975534 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.975514    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"]
Sep 15 17:10:37.978867 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:37.978845    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f299-predictor-576964885f-fklcx"]
Sep 15 17:10:39.448506 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:10:39.448461    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" path="/var/lib/kubelet/pods/8727bf0f-7379-4192-a1f2-c5f1d6ef9e91/volumes"
Sep 15 17:14:30.974323 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:14:30.974271    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:17:57.984262 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:57.984171    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:17:58.047118 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:58.047085    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"]
Sep 15 17:17:58.047374 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:58.047321    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kserve-container" containerID="cri-o://259447bda86c0478ce97f1fbdb5338fa061f7c8adae6026b1781f42b73f7302e" gracePeriod=30
Sep 15 17:17:58.047466 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:58.047346    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kube-rbac-proxy" containerID="cri-o://f6c18b456dbaf760e3827f0fc0eebea4e8832b193ad091ae18f5c35dcd7254e9" gracePeriod=30
Sep 15 17:17:58.068513 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:58.068485    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:17:58.092372 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:58.092343    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"]
Sep 15 17:17:58.092577 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:58.092556    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kserve-container" containerID="cri-o://3fd9860cceb588519d5e2b739b09684927fdea8448959d1e248ce7464231a576" gracePeriod=30
Sep 15 17:17:58.092672 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:58.092574    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kube-rbac-proxy" containerID="cri-o://21ef353105fb4a69034665db254ec4ad5ed64531826bc8c70851aa10ed6beb5e" gracePeriod=30
Sep 15 17:17:58.233837 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:58.233798    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:17:58.454361 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:58.454330    2565 generic.go:356] "Generic (PLEG): container finished" podID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerID="21ef353105fb4a69034665db254ec4ad5ed64531826bc8c70851aa10ed6beb5e" exitCode=2
Sep 15 17:17:58.454361 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:58.454353    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" event={"ID":"ddea253e-be1c-4d28-a8ea-b1c52990edec","Type":"ContainerDied","Data":"21ef353105fb4a69034665db254ec4ad5ed64531826bc8c70851aa10ed6beb5e"}
Sep 15 17:17:58.456103 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:58.456081    2565 generic.go:356] "Generic (PLEG): container finished" podID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerID="f6c18b456dbaf760e3827f0fc0eebea4e8832b193ad091ae18f5c35dcd7254e9" exitCode=2
Sep 15 17:17:58.456213 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:58.456141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" event={"ID":"615f9876-5359-4d83-82cd-56a11fafe4fc","Type":"ContainerDied","Data":"f6c18b456dbaf760e3827f0fc0eebea4e8832b193ad091ae18f5c35dcd7254e9"}
Sep 15 17:17:59.104297 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:59.104238    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:17:59.176110 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:17:59.176058    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:18:01.026448 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.026406    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:18:01.094776 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.094721    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:18:01.130422 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.130392    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jkf2\" (UniqueName: \"kubernetes.io/projected/ddea253e-be1c-4d28-a8ea-b1c52990edec-kube-api-access-9jkf2\") pod \"ddea253e-be1c-4d28-a8ea-b1c52990edec\" (UID: \"ddea253e-be1c-4d28-a8ea-b1c52990edec\") "
Sep 15 17:18:01.130422 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.130423    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddea253e-be1c-4d28-a8ea-b1c52990edec-kube-rbac-proxy-sar-config\") pod \"ddea253e-be1c-4d28-a8ea-b1c52990edec\" (UID: \"ddea253e-be1c-4d28-a8ea-b1c52990edec\") "
Sep 15 17:18:01.130596 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.130455    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddea253e-be1c-4d28-a8ea-b1c52990edec-proxy-tls\") pod \"ddea253e-be1c-4d28-a8ea-b1c52990edec\" (UID: \"ddea253e-be1c-4d28-a8ea-b1c52990edec\") "
Sep 15 17:18:01.130825 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.130795    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddea253e-be1c-4d28-a8ea-b1c52990edec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ddea253e-be1c-4d28-a8ea-b1c52990edec" (UID: "ddea253e-be1c-4d28-a8ea-b1c52990edec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:18:01.132581 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.132556    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddea253e-be1c-4d28-a8ea-b1c52990edec-kube-api-access-9jkf2" (OuterVolumeSpecName: "kube-api-access-9jkf2") pod "ddea253e-be1c-4d28-a8ea-b1c52990edec" (UID: "ddea253e-be1c-4d28-a8ea-b1c52990edec"). InnerVolumeSpecName "kube-api-access-9jkf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:18:01.132687 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.132585    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddea253e-be1c-4d28-a8ea-b1c52990edec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ddea253e-be1c-4d28-a8ea-b1c52990edec" (UID: "ddea253e-be1c-4d28-a8ea-b1c52990edec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:18:01.231168 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.231128    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lznfk\" (UniqueName: \"kubernetes.io/projected/615f9876-5359-4d83-82cd-56a11fafe4fc-kube-api-access-lznfk\") pod \"615f9876-5359-4d83-82cd-56a11fafe4fc\" (UID: \"615f9876-5359-4d83-82cd-56a11fafe4fc\") "
Sep 15 17:18:01.231346 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.231183    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/615f9876-5359-4d83-82cd-56a11fafe4fc-kube-rbac-proxy-sar-config\") pod \"615f9876-5359-4d83-82cd-56a11fafe4fc\" (UID: \"615f9876-5359-4d83-82cd-56a11fafe4fc\") "
Sep 15 17:18:01.231346 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.231213    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/615f9876-5359-4d83-82cd-56a11fafe4fc-proxy-tls\") pod \"615f9876-5359-4d83-82cd-56a11fafe4fc\" (UID: \"615f9876-5359-4d83-82cd-56a11fafe4fc\") "
Sep 15 17:18:01.231471 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.231393    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jkf2\" (UniqueName: \"kubernetes.io/projected/ddea253e-be1c-4d28-a8ea-b1c52990edec-kube-api-access-9jkf2\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:18:01.231471 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.231411    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddea253e-be1c-4d28-a8ea-b1c52990edec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:18:01.231471 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.231424    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddea253e-be1c-4d28-a8ea-b1c52990edec-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:18:01.231571 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.231542    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615f9876-5359-4d83-82cd-56a11fafe4fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "615f9876-5359-4d83-82cd-56a11fafe4fc" (UID: "615f9876-5359-4d83-82cd-56a11fafe4fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:18:01.233195 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.233174    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615f9876-5359-4d83-82cd-56a11fafe4fc-kube-api-access-lznfk" (OuterVolumeSpecName: "kube-api-access-lznfk") pod "615f9876-5359-4d83-82cd-56a11fafe4fc" (UID: "615f9876-5359-4d83-82cd-56a11fafe4fc"). InnerVolumeSpecName "kube-api-access-lznfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:18:01.233261 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.233242    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/615f9876-5359-4d83-82cd-56a11fafe4fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "615f9876-5359-4d83-82cd-56a11fafe4fc" (UID: "615f9876-5359-4d83-82cd-56a11fafe4fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:18:01.332015 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.331979    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/615f9876-5359-4d83-82cd-56a11fafe4fc-proxy-tls\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:18:01.332015 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.332006    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lznfk\" (UniqueName: \"kubernetes.io/projected/615f9876-5359-4d83-82cd-56a11fafe4fc-kube-api-access-lznfk\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:18:01.332015 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.332016    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/615f9876-5359-4d83-82cd-56a11fafe4fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-77.ec2.internal\" DevicePath \"\""
Sep 15 17:18:01.467776 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.467741    2565 generic.go:356] "Generic (PLEG): container finished" podID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerID="3fd9860cceb588519d5e2b739b09684927fdea8448959d1e248ce7464231a576" exitCode=0
Sep 15 17:18:01.467917 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.467794    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"
Sep 15 17:18:01.467917 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.467823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" event={"ID":"ddea253e-be1c-4d28-a8ea-b1c52990edec","Type":"ContainerDied","Data":"3fd9860cceb588519d5e2b739b09684927fdea8448959d1e248ce7464231a576"}
Sep 15 17:18:01.467917 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.467883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch" event={"ID":"ddea253e-be1c-4d28-a8ea-b1c52990edec","Type":"ContainerDied","Data":"167066463ac0d49d6b8c6a15031244b783e02c187805d00833a3784e472db977"}
Sep 15 17:18:01.468088 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.467920    2565 scope.go:117] "RemoveContainer" containerID="21ef353105fb4a69034665db254ec4ad5ed64531826bc8c70851aa10ed6beb5e"
Sep 15 17:18:01.469469 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.469428    2565 generic.go:356] "Generic (PLEG): container finished" podID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerID="259447bda86c0478ce97f1fbdb5338fa061f7c8adae6026b1781f42b73f7302e" exitCode=0
Sep 15 17:18:01.469601 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.469513    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" event={"ID":"615f9876-5359-4d83-82cd-56a11fafe4fc","Type":"ContainerDied","Data":"259447bda86c0478ce97f1fbdb5338fa061f7c8adae6026b1781f42b73f7302e"}
Sep 15 17:18:01.469601 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.469562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9" event={"ID":"615f9876-5359-4d83-82cd-56a11fafe4fc","Type":"ContainerDied","Data":"0d2c95da3b116a70a7f89d40aba34827a924402694c7fdfd9768f074c333e8b9"}
Sep 15 17:18:01.469753 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.469606    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"
Sep 15 17:18:01.475925 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.475905    2565 scope.go:117] "RemoveContainer" containerID="3fd9860cceb588519d5e2b739b09684927fdea8448959d1e248ce7464231a576"
Sep 15 17:18:01.482926 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.482901    2565 scope.go:117] "RemoveContainer" containerID="21ef353105fb4a69034665db254ec4ad5ed64531826bc8c70851aa10ed6beb5e"
Sep 15 17:18:01.483241 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:18:01.483216    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ef353105fb4a69034665db254ec4ad5ed64531826bc8c70851aa10ed6beb5e\": container with ID starting with 21ef353105fb4a69034665db254ec4ad5ed64531826bc8c70851aa10ed6beb5e not found: ID does not exist" containerID="21ef353105fb4a69034665db254ec4ad5ed64531826bc8c70851aa10ed6beb5e"
Sep 15 17:18:01.483296 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.483251    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ef353105fb4a69034665db254ec4ad5ed64531826bc8c70851aa10ed6beb5e"} err="failed to get container status \"21ef353105fb4a69034665db254ec4ad5ed64531826bc8c70851aa10ed6beb5e\": rpc error: code = NotFound desc = could not find container \"21ef353105fb4a69034665db254ec4ad5ed64531826bc8c70851aa10ed6beb5e\": container with ID starting with 21ef353105fb4a69034665db254ec4ad5ed64531826bc8c70851aa10ed6beb5e not found: ID does not exist"
Sep 15 17:18:01.483296 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.483274    2565 scope.go:117] "RemoveContainer" containerID="3fd9860cceb588519d5e2b739b09684927fdea8448959d1e248ce7464231a576"
Sep 15 17:18:01.483584 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:18:01.483557    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd9860cceb588519d5e2b739b09684927fdea8448959d1e248ce7464231a576\": container with ID starting with 3fd9860cceb588519d5e2b739b09684927fdea8448959d1e248ce7464231a576 not found: ID does not exist" containerID="3fd9860cceb588519d5e2b739b09684927fdea8448959d1e248ce7464231a576"
Sep 15 17:18:01.483672 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.483584    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd9860cceb588519d5e2b739b09684927fdea8448959d1e248ce7464231a576"} err="failed to get container status \"3fd9860cceb588519d5e2b739b09684927fdea8448959d1e248ce7464231a576\": rpc error: code = NotFound desc = could not find container \"3fd9860cceb588519d5e2b739b09684927fdea8448959d1e248ce7464231a576\": container with ID starting with 3fd9860cceb588519d5e2b739b09684927fdea8448959d1e248ce7464231a576 not found: ID does not exist"
Sep 15 17:18:01.483672 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.483600    2565 scope.go:117] "RemoveContainer" containerID="f6c18b456dbaf760e3827f0fc0eebea4e8832b193ad091ae18f5c35dcd7254e9"
Sep 15 17:18:01.483847 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.483826    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"]
Sep 15 17:18:01.487558 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.487539    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-969ee-predictor-7bffb9f7df-jdmch"]
Sep 15 17:18:01.490607 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.490592    2565 scope.go:117] "RemoveContainer" containerID="259447bda86c0478ce97f1fbdb5338fa061f7c8adae6026b1781f42b73f7302e"
Sep 15 17:18:01.497130 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.497112    2565 scope.go:117] "RemoveContainer" containerID="f6c18b456dbaf760e3827f0fc0eebea4e8832b193ad091ae18f5c35dcd7254e9"
Sep 15 17:18:01.497375 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:18:01.497355    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c18b456dbaf760e3827f0fc0eebea4e8832b193ad091ae18f5c35dcd7254e9\": container with ID starting with f6c18b456dbaf760e3827f0fc0eebea4e8832b193ad091ae18f5c35dcd7254e9 not found: ID does not exist" containerID="f6c18b456dbaf760e3827f0fc0eebea4e8832b193ad091ae18f5c35dcd7254e9"
Sep 15 17:18:01.497448 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.497380    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c18b456dbaf760e3827f0fc0eebea4e8832b193ad091ae18f5c35dcd7254e9"} err="failed to get container status \"f6c18b456dbaf760e3827f0fc0eebea4e8832b193ad091ae18f5c35dcd7254e9\": rpc error: code = NotFound desc = could not find container \"f6c18b456dbaf760e3827f0fc0eebea4e8832b193ad091ae18f5c35dcd7254e9\": container with ID starting with f6c18b456dbaf760e3827f0fc0eebea4e8832b193ad091ae18f5c35dcd7254e9 not found: ID does not exist"
Sep 15 17:18:01.497448 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.497389    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"]
Sep 15 17:18:01.497448 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.497396    2565 scope.go:117] "RemoveContainer" containerID="259447bda86c0478ce97f1fbdb5338fa061f7c8adae6026b1781f42b73f7302e"
Sep 15 17:18:01.497679 ip-10-0-134-77 kubenswrapper[2565]: E0915 17:18:01.497664    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259447bda86c0478ce97f1fbdb5338fa061f7c8adae6026b1781f42b73f7302e\": container with ID starting with 259447bda86c0478ce97f1fbdb5338fa061f7c8adae6026b1781f42b73f7302e not found: ID does not exist" containerID="259447bda86c0478ce97f1fbdb5338fa061f7c8adae6026b1781f42b73f7302e"
Sep 15 17:18:01.497752 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.497682    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259447bda86c0478ce97f1fbdb5338fa061f7c8adae6026b1781f42b73f7302e"} err="failed to get container status \"259447bda86c0478ce97f1fbdb5338fa061f7c8adae6026b1781f42b73f7302e\": rpc error: code = NotFound desc = could not find container \"259447bda86c0478ce97f1fbdb5338fa061f7c8adae6026b1781f42b73f7302e\": container with ID starting with 259447bda86c0478ce97f1fbdb5338fa061f7c8adae6026b1781f42b73f7302e not found: ID does not exist"
Sep 15 17:18:01.499224 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:01.499204    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-969ee-predictor-84b4498f4d-sfpj9"]
Sep 15 17:18:03.448061 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:03.448025    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" path="/var/lib/kubelet/pods/615f9876-5359-4d83-82cd-56a11fafe4fc/volumes"
Sep 15 17:18:03.448501 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:03.448431    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" path="/var/lib/kubelet/pods/ddea253e-be1c-4d28-a8ea-b1c52990edec/volumes"
Sep 15 17:18:30.324470 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.324432    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:18:30.455730 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.455675    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t4w98/must-gather-pqrkr"]
Sep 15 17:18:30.456010 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.455992    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456091 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456012    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456091 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456029    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kserve-container"
Sep 15 17:18:30.456091 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456037    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kserve-container"
Sep 15 17:18:30.456091 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456049    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456091 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456059    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456091 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456072    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456091 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456081    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456097    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456105    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456115    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456125    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456134    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456145    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456155    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456163    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456178    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456186    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456198    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456206    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456215    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456224    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456234    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456243    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456312    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456324    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456334    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456345    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456354    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456363    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b429c763-f770-4b6e-a198-966171a160e0" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456371    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8727bf0f-7379-4192-a1f2-c5f1d6ef9e91" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456379    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kserve-container"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456386    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="615f9876-5359-4d83-82cd-56a11fafe4fc" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456394    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddea253e-be1c-4d28-a8ea-b1c52990edec" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456402    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="04de9312-8a16-4824-ac77-fad3abef545a" containerName="kube-rbac-proxy"
Sep 15 17:18:30.456459 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.456410    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0365941-c04c-4a07-bed6-c012cba8189b" containerName="kube-rbac-proxy"
Sep 15 17:18:30.461145 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.461124    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4w98/must-gather-pqrkr"
Sep 15 17:18:30.463757 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.463730    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t4w98\"/\"default-dockercfg-98xd7\""
Sep 15 17:18:30.464793 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.464776    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t4w98\"/\"kube-root-ca.crt\""
Sep 15 17:18:30.464891 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.464805    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t4w98\"/\"openshift-service-ca.crt\""
Sep 15 17:18:30.467129 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.467108    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4w98/must-gather-pqrkr"]
Sep 15 17:18:30.469996 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.469966    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:18:30.528920 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.528887    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:18:30.533869 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.533851    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/71cfd61f-3238-46e2-9b65-9dd3c2f51c84-must-gather-output\") pod \"must-gather-pqrkr\" (UID: \"71cfd61f-3238-46e2-9b65-9dd3c2f51c84\") " pod="openshift-must-gather-t4w98/must-gather-pqrkr"
Sep 15 17:18:30.533933 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.533884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tjvm\" (UniqueName: \"kubernetes.io/projected/71cfd61f-3238-46e2-9b65-9dd3c2f51c84-kube-api-access-7tjvm\") pod \"must-gather-pqrkr\" (UID: \"71cfd61f-3238-46e2-9b65-9dd3c2f51c84\") " pod="openshift-must-gather-t4w98/must-gather-pqrkr"
Sep 15 17:18:30.635034 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.634947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/71cfd61f-3238-46e2-9b65-9dd3c2f51c84-must-gather-output\") pod \"must-gather-pqrkr\" (UID: \"71cfd61f-3238-46e2-9b65-9dd3c2f51c84\") " pod="openshift-must-gather-t4w98/must-gather-pqrkr"
Sep 15 17:18:30.635034 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.634985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tjvm\" (UniqueName: \"kubernetes.io/projected/71cfd61f-3238-46e2-9b65-9dd3c2f51c84-kube-api-access-7tjvm\") pod \"must-gather-pqrkr\" (UID: \"71cfd61f-3238-46e2-9b65-9dd3c2f51c84\") " pod="openshift-must-gather-t4w98/must-gather-pqrkr"
Sep 15 17:18:30.635343 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.635323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/71cfd61f-3238-46e2-9b65-9dd3c2f51c84-must-gather-output\") pod \"must-gather-pqrkr\" (UID: \"71cfd61f-3238-46e2-9b65-9dd3c2f51c84\") " pod="openshift-must-gather-t4w98/must-gather-pqrkr"
Sep 15 17:18:30.643195 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.643172    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tjvm\" (UniqueName: \"kubernetes.io/projected/71cfd61f-3238-46e2-9b65-9dd3c2f51c84-kube-api-access-7tjvm\") pod \"must-gather-pqrkr\" (UID: \"71cfd61f-3238-46e2-9b65-9dd3c2f51c84\") " pod="openshift-must-gather-t4w98/must-gather-pqrkr"
Sep 15 17:18:30.771642 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:30.771604    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4w98/must-gather-pqrkr"
Sep 15 17:18:31.038988 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:31.038894    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4w98/must-gather-pqrkr"]
Sep 15 17:18:31.041716 ip-10-0-134-77 kubenswrapper[2565]: W0915 17:18:31.041677    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71cfd61f_3238_46e2_9b65_9dd3c2f51c84.slice/crio-fa5787567a61002a4b5f1b72731c62d1568651ddb54ce7aeb6099a16a0ba8304 WatchSource:0}: Error finding container fa5787567a61002a4b5f1b72731c62d1568651ddb54ce7aeb6099a16a0ba8304: Status 404 returned error can't find the container with id fa5787567a61002a4b5f1b72731c62d1568651ddb54ce7aeb6099a16a0ba8304
Sep 15 17:18:31.044050 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:31.044033    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 17:18:31.571401 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:31.571366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4w98/must-gather-pqrkr" event={"ID":"71cfd61f-3238-46e2-9b65-9dd3c2f51c84","Type":"ContainerStarted","Data":"fa5787567a61002a4b5f1b72731c62d1568651ddb54ce7aeb6099a16a0ba8304"}
Sep 15 17:18:32.579088 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:32.579050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4w98/must-gather-pqrkr" event={"ID":"71cfd61f-3238-46e2-9b65-9dd3c2f51c84","Type":"ContainerStarted","Data":"c9aa04d8c5e10703a0200aa77fc4cbe845e2a251391f8f579ad9672675bbd250"}
Sep 15 17:18:32.579952 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:32.579627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4w98/must-gather-pqrkr" event={"ID":"71cfd61f-3238-46e2-9b65-9dd3c2f51c84","Type":"ContainerStarted","Data":"fffbc39a174ddb6ff88ab21c5ce63c2061d7ca56814696529b8591d3f9d076db"}
Sep 15 17:18:32.595962 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:32.595905    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t4w98/must-gather-pqrkr" podStartSLOduration=1.735300142 podStartE2EDuration="2.595886607s" totalImagesPullingTime="860.586465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:18:32.595823501 +0000 UTC m=+3103.640209230" watchObservedRunningTime="2026-09-15 17:18:32.595886607 +0000 UTC m=+3103.640272341"
Sep 15 17:18:33.290515 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:33.290484    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jvr5t_22495e12-7c88-4178-9d83-076062a2c424/global-pull-secret-syncer/0.log"
Sep 15 17:18:33.377756 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:33.377723    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log"
Sep 15 17:18:33.485452 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:33.485421    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-77.ec2.internal_ce9c1ce3a3fab75c3e888d6b8fff341b/haproxy/0.log"
Sep 15 17:18:36.344683 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:36.344642    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:18:36.434570 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:36.434505    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fjwgp_15e6809f-8f4b-4eeb-ba75-9ee8226ad2da/konnectivity-agent/0.log" limit=1024
Sep 15 17:18:37.662949 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:37.662914    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gjj84_e2e585b9-a509-4241-b3f0-7432f5633b21/node-exporter/0.log"
Sep 15 17:18:37.684618 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:37.684583    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gjj84_e2e585b9-a509-4241-b3f0-7432f5633b21/kube-rbac-proxy/0.log"
Sep 15 17:18:37.711216 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:37.711188    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gjj84_e2e585b9-a509-4241-b3f0-7432f5633b21/init-textfile/0.log"
Sep 15 17:18:41.864360 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:41.864326    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"]
Sep 15 17:18:41.869437 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:41.869411    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:41.875653 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:41.875621    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"]
Sep 15 17:18:41.917217 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:41.917141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bf177b8-818c-451b-8596-bd536b9d08f0-sys\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:41.917398 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:41.917231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9bf177b8-818c-451b-8596-bd536b9d08f0-podres\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:41.917398 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:41.917379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9bf177b8-818c-451b-8596-bd536b9d08f0-proc\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:41.917520 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:41.917430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5n9g\" (UniqueName: \"kubernetes.io/projected/9bf177b8-818c-451b-8596-bd536b9d08f0-kube-api-access-h5n9g\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:41.917520 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:41.917509    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bf177b8-818c-451b-8596-bd536b9d08f0-lib-modules\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:42.017975 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.017938    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9bf177b8-818c-451b-8596-bd536b9d08f0-podres\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:42.018277 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.018237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9bf177b8-818c-451b-8596-bd536b9d08f0-proc\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:42.018437 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.018421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5n9g\" (UniqueName: \"kubernetes.io/projected/9bf177b8-818c-451b-8596-bd536b9d08f0-kube-api-access-h5n9g\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:42.018595 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.018571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bf177b8-818c-451b-8596-bd536b9d08f0-lib-modules\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:42.018595 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.018597    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9bf177b8-818c-451b-8596-bd536b9d08f0-proc\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:42.018821 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.018637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bf177b8-818c-451b-8596-bd536b9d08f0-sys\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:42.018821 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.018652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9bf177b8-818c-451b-8596-bd536b9d08f0-podres\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:42.018821 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.018777    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bf177b8-818c-451b-8596-bd536b9d08f0-sys\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:42.018992 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.018840    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bf177b8-818c-451b-8596-bd536b9d08f0-lib-modules\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:42.026027 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.026002    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5n9g\" (UniqueName: \"kubernetes.io/projected/9bf177b8-818c-451b-8596-bd536b9d08f0-kube-api-access-h5n9g\") pod \"perf-node-gather-daemonset-dc4ck\" (UID: \"9bf177b8-818c-451b-8596-bd536b9d08f0\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:42.128145 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.128072    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-czx4f_ef4b4ee7-812d-4ed4-9225-c2f5616c8133/dns/0.log"
Sep 15 17:18:42.148802 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.148775    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-czx4f_ef4b4ee7-812d-4ed4-9225-c2f5616c8133/kube-rbac-proxy/0.log"
Sep 15 17:18:42.183151 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.183110    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:42.259970 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.259947    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mj466_b7764505-fade-42cf-9a24-191b27545755/dns-node-resolver/0.log"
Sep 15 17:18:42.459879 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.459846    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"]
Sep 15 17:18:42.462861 ip-10-0-134-77 kubenswrapper[2565]: W0915 17:18:42.462834    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9bf177b8_818c_451b_8596_bd536b9d08f0.slice/crio-4c0463468324a07cae85054e156e6acf0151092536c452bcaccf638809cc24c6 WatchSource:0}: Error finding container 4c0463468324a07cae85054e156e6acf0151092536c452bcaccf638809cc24c6: Status 404 returned error can't find the container with id 4c0463468324a07cae85054e156e6acf0151092536c452bcaccf638809cc24c6
Sep 15 17:18:42.640830 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.640784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck" event={"ID":"9bf177b8-818c-451b-8596-bd536b9d08f0","Type":"ContainerStarted","Data":"55bc5e6a9d22e89e40a47af47564e2971b72888f23d09d53f18aff0a34328094"}
Sep 15 17:18:42.641005 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.640860    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck" event={"ID":"9bf177b8-818c-451b-8596-bd536b9d08f0","Type":"ContainerStarted","Data":"4c0463468324a07cae85054e156e6acf0151092536c452bcaccf638809cc24c6"}
Sep 15 17:18:42.641005 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.640892    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:42.656358 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.656300    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck" podStartSLOduration=1.6562801459999998 podStartE2EDuration="1.656280146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 17:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:18:42.656130534 +0000 UTC m=+3113.700516267" watchObservedRunningTime="2026-09-15 17:18:42.656280146 +0000 UTC m=+3113.700665879"
Sep 15 17:18:42.792968 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:42.792891    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7np85_5a34bc31-c8e6-4214-9762-366d6fe23190/node-ca/0.log"
Sep 15 17:18:43.965320 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:43.965289    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kdqvv_aaa40fc5-09b9-4768-95b9-005e46e994ed/serve-healthcheck-canary/0.log"
Sep 15 17:18:44.471054 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:44.471024    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mwbnk_148e3028-957d-460a-abfe-e936406d0989/kube-rbac-proxy/0.log"
Sep 15 17:18:44.491873 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:44.491846    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mwbnk_148e3028-957d-460a-abfe-e936406d0989/exporter/0.log"
Sep 15 17:18:44.512927 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:44.512900    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mwbnk_148e3028-957d-460a-abfe-e936406d0989/extractor/0.log"
Sep 15 17:18:46.739731 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:46.739690    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-z9sdp_b4ec8569-263e-4bea-bd51-7b5df4cfde37/manager/0.log"
Sep 15 17:18:48.656899 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:48.656867    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-dc4ck"
Sep 15 17:18:51.027859 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:51.027809    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-d4pf4_2bd47ace-dade-4b8d-af86-a27937a4ac38/migrator/0.log"
Sep 15 17:18:51.050270 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:51.050239    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-d4pf4_2bd47ace-dade-4b8d-af86-a27937a4ac38/graceful-termination/0.log"
Sep 15 17:18:52.510310 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:52.510219    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxfh_150d8a61-545c-417a-b159-f2e84473250a/kube-multus-additional-cni-plugins/0.log"
Sep 15 17:18:52.532332 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:52.532306    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxfh_150d8a61-545c-417a-b159-f2e84473250a/egress-router-binary-copy/0.log"
Sep 15 17:18:52.553378 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:52.553356    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxfh_150d8a61-545c-417a-b159-f2e84473250a/cni-plugins/0.log"
Sep 15 17:18:52.573986 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:52.573963    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxfh_150d8a61-545c-417a-b159-f2e84473250a/bond-cni-plugin/0.log"
Sep 15 17:18:52.595816 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:52.595794    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxfh_150d8a61-545c-417a-b159-f2e84473250a/routeoverride-cni/0.log"
Sep 15 17:18:52.616916 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:52.616887    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxfh_150d8a61-545c-417a-b159-f2e84473250a/whereabouts-cni-bincopy/0.log"
Sep 15 17:18:52.639153 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:52.639125    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxxfh_150d8a61-545c-417a-b159-f2e84473250a/whereabouts-cni/0.log"
Sep 15 17:18:52.881798 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:52.881764    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-txbkb_2f7d621a-2df5-4eb9-b76c-a8e244869670/kube-multus/0.log"
Sep 15 17:18:52.980667 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:52.980627    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nvl8x_9a34ad6e-33f8-4f96-9aca-98998f3bcd62/network-metrics-daemon/0.log"
Sep 15 17:18:53.003272 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:53.003245    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nvl8x_9a34ad6e-33f8-4f96-9aca-98998f3bcd62/kube-rbac-proxy/0.log"
Sep 15 17:18:53.778297 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:53.778264    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bwpd_ac156fa7-e35f-4265-baa3-be8f567a8190/ovn-controller/0.log"
Sep 15 17:18:53.795471 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:53.795448    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bwpd_ac156fa7-e35f-4265-baa3-be8f567a8190/ovn-acl-logging/0.log"
Sep 15 17:18:53.817000 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:53.816977    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bwpd_ac156fa7-e35f-4265-baa3-be8f567a8190/kube-rbac-proxy-node/0.log"
Sep 15 17:18:53.838644 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:53.838620    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bwpd_ac156fa7-e35f-4265-baa3-be8f567a8190/kube-rbac-proxy-ovn-metrics/0.log"
Sep 15 17:18:53.859452 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:53.859425    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bwpd_ac156fa7-e35f-4265-baa3-be8f567a8190/northd/0.log"
Sep 15 17:18:53.880867 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:53.880844    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bwpd_ac156fa7-e35f-4265-baa3-be8f567a8190/nbdb/0.log"
Sep 15 17:18:53.900762 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:53.900736    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bwpd_ac156fa7-e35f-4265-baa3-be8f567a8190/sbdb/0.log"
Sep 15 17:18:53.944816 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:53.944787    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bwpd_ac156fa7-e35f-4265-baa3-be8f567a8190/ovnkube-controller/0.log"
Sep 15 17:18:55.386155 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:55.386126    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-vjsq5_33198472-1392-416c-a639-0ba97f72fd7f/network-check-target-container/0.log"
Sep 15 17:18:56.318524 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:56.318479    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-kws86_79a248eb-edf5-486b-b941-217db1abc9ea/iptables-alerter/0.log"
Sep 15 17:18:56.962097 ip-10-0-134-77 kubenswrapper[2565]: I0915 17:18:56.962067    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-mnwxq_32b7532a-7fdd-4341-881f-2715c75f2b88/tuned/0.log"